{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 动量法\n",
    "\n",
    "在[“梯度下降和随机梯度下降”](./gd-sgd.ipynb)一节中我们提到，目标函数有关自变量的梯度代表了目标函数在自变量当前位置下降最快的方向。因此，梯度下降也叫作最陡下降（steepest descent）。在每次迭代中，梯度下降根据自变量当前位置，沿着当前位置的梯度更新自变量。然而，如果自变量的迭代方向仅仅取决于自变量当前位置，这可能会带来一些问题。\n",
    "\n",
    "\n",
    "## 梯度下降的问题\n",
    "\n",
    "让我们考虑一个输入和输出分别为二维向量$\\boldsymbol{x} = [x_1, x_2]^\\top$和标量的目标函数$f(\\boldsymbol{x})=0.1x_1^2+2x_2^2$。与[“梯度下降和随机梯度下降”](./gd-sgd.ipynb)一节中不同，这里将$x_1^2$系数从$1$减小到了$0.1$。下面实现基于这个目标函数的梯度下降，并演示使用学习率为$0.4$时自变量的迭代轨迹。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "3"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 20, x1 -0.943467, x2 -0.000073\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 248.620313 184.15625\" width=\"248.620313pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 184.15625 \n",
       "L 248.620313 184.15625 \n",
       "L 248.620313 -0 \n",
       "L 0 -0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 42.620312 146.6 \n",
       "L 237.920313 146.6 \n",
       "L 237.920313 10.7 \n",
       "L 42.620312 10.7 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"m26882394e2\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"88.39375\" xlink:href=\"#m26882394e2\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −4 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-2212\"/>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"DejaVuSans-34\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(81.022656 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"149.425\" xlink:href=\"#m26882394e2\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −2 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-32\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(142.053906 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"210.45625\" xlink:href=\"#m26882394e2\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-30\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(207.275 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_4\">\n",
       "     <!-- x1 -->\n",
       "     <defs>\n",
       "      <path d=\"M 54.890625 54.6875 \n",
       "L 35.109375 28.078125 \n",
       "L 55.90625 0 \n",
       "L 45.3125 0 \n",
       "L 29.390625 21.484375 \n",
       "L 13.484375 0 \n",
       "L 2.875 0 \n",
       "L 24.125 28.609375 \n",
       "L 4.6875 54.6875 \n",
       "L 15.28125 54.6875 \n",
       "L 29.78125 35.203125 \n",
       "L 44.28125 54.6875 \n",
       "z\n",
       "\" id=\"DejaVuSans-78\"/>\n",
       "      <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-31\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(134.129687 174.876562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "      <use x=\"59.179688\" xlink:href=\"#DejaVuSans-31\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"mcd3c04b751\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620312\" xlink:href=\"#mcd3c04b751\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- −3 -->\n",
       "      <defs>\n",
       "       <path d=\"M 40.578125 39.3125 \n",
       "Q 47.65625 37.796875 51.625 33 \n",
       "Q 55.609375 28.21875 55.609375 21.1875 \n",
       "Q 55.609375 10.40625 48.1875 4.484375 \n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \n",
       "Q 12.796875 0.390625 7.625 2.203125 \n",
       "L 7.625 11.71875 \n",
       "Q 11.71875 9.328125 16.59375 8.109375 \n",
       "Q 21.484375 6.890625 26.8125 6.890625 \n",
       "Q 36.078125 6.890625 40.9375 10.546875 \n",
       "Q 45.796875 14.203125 45.796875 21.1875 \n",
       "Q 45.796875 27.640625 41.28125 31.265625 \n",
       "Q 36.765625 34.90625 28.71875 34.90625 \n",
       "L 20.21875 34.90625 \n",
       "L 20.21875 43.015625 \n",
       "L 29.109375 43.015625 \n",
       "Q 36.375 43.015625 40.234375 45.921875 \n",
       "Q 44.09375 48.828125 44.09375 54.296875 \n",
       "Q 44.09375 59.90625 40.109375 62.90625 \n",
       "Q 36.140625 65.921875 28.71875 65.921875 \n",
       "Q 24.65625 65.921875 20.015625 65.03125 \n",
       "Q 15.375 64.15625 9.8125 62.3125 \n",
       "L 9.8125 71.09375 \n",
       "Q 15.4375 72.65625 20.34375 73.4375 \n",
       "Q 25.25 74.21875 29.59375 74.21875 \n",
       "Q 40.828125 74.21875 47.359375 69.109375 \n",
       "Q 53.90625 64.015625 53.90625 55.328125 \n",
       "Q 53.90625 49.265625 50.4375 45.09375 \n",
       "Q 46.96875 40.921875 40.578125 39.3125 \n",
       "z\n",
       "\" id=\"DejaVuSans-33\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(20.878125 150.399219)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620312\" xlink:href=\"#mcd3c04b751\" y=\"119.952941\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- −2 -->\n",
       "      <g transform=\"translate(20.878125 123.75216)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620312\" xlink:href=\"#mcd3c04b751\" y=\"93.305882\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- −1 -->\n",
       "      <g transform=\"translate(20.878125 97.105101)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-31\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620312\" xlink:href=\"#mcd3c04b751\" y=\"66.658824\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(29.257812 70.458042)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620312\" xlink:href=\"#mcd3c04b751\" y=\"40.011765\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 1 -->\n",
       "      <g transform=\"translate(29.257812 43.810983)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620312\" xlink:href=\"#mcd3c04b751\" y=\"13.364706\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(29.257812 17.163925)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- x2 -->\n",
       "     <g transform=\"translate(14.798437 84.790625)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "      <use x=\"59.179688\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_1\">\n",
       "    <path clip-path=\"url(#pd103e112b8)\" d=\"M 237.920313 98.843141 \n",
       "L 222.6625 99.189553 \n",
       "L 207.404688 99.269494 \n",
       "L 192.146875 99.082965 \n",
       "L 176.889063 98.629501 \n",
       "L 164.682813 98.027046 \n",
       "L 152.476563 97.23922 \n",
       "L 140.270313 96.266023 \n",
       "L 131.115625 95.361513 \n",
       "L 118.909375 93.940336 \n",
       "L 109.754688 92.68178 \n",
       "L 97.100875 90.641176 \n",
       "L 85.342188 88.282128 \n",
       "L 82.290625 87.585647 \n",
       "L 73.135938 85.26052 \n",
       "L 63.98125 82.356364 \n",
       "L 57.878125 79.982353 \n",
       "L 52.486044 77.317647 \n",
       "L 48.409725 74.652941 \n",
       "L 45.559891 71.988235 \n",
       "L 43.880132 69.323529 \n",
       "L 43.320212 66.658824 \n",
       "L 43.880132 63.994118 \n",
       "L 45.671875 61.222824 \n",
       "L 48.723438 58.455336 \n",
       "L 52.486044 56 \n",
       "L 57.878125 53.335294 \n",
       "L 64.752171 50.670588 \n",
       "L 73.307374 48.005882 \n",
       "L 83.908321 45.341176 \n",
       "L 94.496875 43.178062 \n",
       "L 100.6 42.08743 \n",
       "L 113.96875 40.011765 \n",
       "L 125.0125 38.641345 \n",
       "L 137.21875 37.323887 \n",
       "L 149.425 36.304348 \n",
       "L 161.63125 35.470179 \n",
       "L 173.8375 34.821381 \n",
       "L 186.04375 34.383906 \n",
       "L 201.301563 34.090788 \n",
       "L 216.559375 34.064141 \n",
       "L 231.817188 34.303965 \n",
       "L 237.920313 34.474506 \n",
       "L 237.920313 34.474506 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_2\">\n",
       "    <path clip-path=\"url(#pd103e112b8)\" d=\"M 237.920313 112.487958 \n",
       "L 234.86875 112.552672 \n",
       "L 231.817188 112.609773 \n",
       "L 228.765625 112.659261 \n",
       "L 225.714063 112.701134 \n",
       "L 222.6625 112.735395 \n",
       "L 219.610938 112.762042 \n",
       "L 216.559375 112.781076 \n",
       "L 213.507813 112.792496 \n",
       "L 210.45625 112.796303 \n",
       "L 207.404688 112.792496 \n",
       "L 204.353125 112.781076 \n",
       "L 201.301563 112.762042 \n",
       "L 198.25 112.735395 \n",
       "L 195.198438 112.701134 \n",
       "L 192.146875 112.659261 \n",
       "L 189.095313 112.609773 \n",
       "L 186.04375 112.552672 \n",
       "L 182.992188 112.487958 \n",
       "L 179.940625 112.41563 \n",
       "L 176.889063 112.335689 \n",
       "L 173.8375 112.248134 \n",
       "L 170.785938 112.152966 \n",
       "L 167.734375 112.050185 \n",
       "L 165.208944 111.958824 \n",
       "L 164.682813 111.938636 \n",
       "L 161.63125 111.813476 \n",
       "L 158.579688 111.680241 \n",
       "L 155.528125 111.53893 \n",
       "L 152.476563 111.389545 \n",
       "L 149.425 111.232086 \n",
       "L 146.373438 111.066551 \n",
       "L 143.321875 110.892941 \n",
       "L 140.270313 110.711257 \n",
       "L 137.21875 110.521497 \n",
       "L 134.167188 110.323663 \n",
       "L 131.115625 110.117754 \n",
       "L 128.064063 109.90377 \n",
       "L 125.0125 109.681711 \n",
       "L 121.960938 109.451578 \n",
       "L 119.943803 109.294118 \n",
       "L 118.909375 109.208159 \n",
       "L 115.857813 108.945987 \n",
       "L 112.80625 108.675218 \n",
       "L 109.754688 108.395854 \n",
       "L 106.703125 108.107894 \n",
       "L 103.651563 107.811338 \n",
       "L 100.6 107.506186 \n",
       "L 97.548438 107.192438 \n",
       "L 94.496875 106.870095 \n",
       "L 92.277557 106.629412 \n",
       "L 91.445313 106.532931 \n",
       "L 88.39375 106.16998 \n",
       "L 85.342188 105.79784 \n",
       "L 82.290625 105.416511 \n",
       "L 79.239063 105.025994 \n",
       "L 76.1875 104.626288 \n",
       "L 73.135938 104.217394 \n",
       "L 71.291587 103.964706 \n",
       "L 70.084375 103.787059 \n",
       "L 67.032813 103.328137 \n",
       "L 63.98125 102.859346 \n",
       "L 60.929688 102.380686 \n",
       "L 57.878125 101.892157 \n",
       "L 54.826563 101.393758 \n",
       "L 54.263653 101.3 \n",
       "L 51.775 100.852329 \n",
       "L 48.723438 100.292741 \n",
       "L 45.671875 99.722494 \n",
       "L 42.620313 99.141588 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pd103e112b8)\" d=\"M 42.620313 34.176059 \n",
       "L 45.671875 33.595153 \n",
       "L 48.723438 33.024906 \n",
       "L 51.775 32.465318 \n",
       "L 54.263653 32.017647 \n",
       "L 54.826563 31.923889 \n",
       "L 57.878125 31.42549 \n",
       "L 60.929688 30.936961 \n",
       "L 63.98125 30.458301 \n",
       "L 67.032813 29.98951 \n",
       "L 70.084375 29.530588 \n",
       "L 71.291587 29.352941 \n",
       "L 73.135938 29.100254 \n",
       "L 76.1875 28.691359 \n",
       "L 79.239063 28.291653 \n",
       "L 82.290625 27.901136 \n",
       "L 85.342188 27.519807 \n",
       "L 88.39375 27.147667 \n",
       "L 91.445313 26.784716 \n",
       "L 92.277557 26.688235 \n",
       "L 94.496875 26.447552 \n",
       "L 97.548438 26.125209 \n",
       "L 100.6 25.811461 \n",
       "L 103.651563 25.506309 \n",
       "L 106.703125 25.209753 \n",
       "L 109.754688 24.921793 \n",
       "L 112.80625 24.642429 \n",
       "L 115.857813 24.37166 \n",
       "L 118.909375 24.109488 \n",
       "L 119.943803 24.023529 \n",
       "L 121.960938 23.86607 \n",
       "L 125.0125 23.635936 \n",
       "L 128.064063 23.413877 \n",
       "L 131.115625 23.199893 \n",
       "L 134.167188 22.993984 \n",
       "L 137.21875 22.79615 \n",
       "L 140.270313 22.60639 \n",
       "L 143.321875 22.424706 \n",
       "L 146.373438 22.251096 \n",
       "L 149.425 22.085561 \n",
       "L 152.476563 21.928102 \n",
       "L 155.528125 21.778717 \n",
       "L 158.579688 21.637406 \n",
       "L 161.63125 21.504171 \n",
       "L 164.682813 21.379011 \n",
       "L 165.208944 21.358824 \n",
       "L 167.734375 21.267462 \n",
       "L 170.785938 21.164681 \n",
       "L 173.8375 21.069513 \n",
       "L 176.889063 20.981958 \n",
       "L 179.940625 20.902017 \n",
       "L 182.992188 20.829689 \n",
       "L 186.04375 20.764975 \n",
       "L 189.095313 20.707874 \n",
       "L 192.146875 20.658387 \n",
       "L 195.198438 20.616513 \n",
       "L 198.25 20.582252 \n",
       "L 201.301563 20.555605 \n",
       "L 204.353125 20.536571 \n",
       "L 207.404688 20.525151 \n",
       "L 210.45625 20.521345 \n",
       "L 213.507813 20.525151 \n",
       "L 216.559375 20.536571 \n",
       "L 219.610938 20.555605 \n",
       "L 222.6625 20.582252 \n",
       "L 225.714063 20.616513 \n",
       "L 228.765625 20.658387 \n",
       "L 231.817188 20.707874 \n",
       "L 234.86875 20.764975 \n",
       "L 237.920313 20.829689 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_3\">\n",
       "    <path clip-path=\"url(#pd103e112b8)\" d=\"M 237.920313 122.924398 \n",
       "L 234.86875 122.977073 \n",
       "L 231.817188 123.02355 \n",
       "L 228.765625 123.06383 \n",
       "L 225.714063 123.097914 \n",
       "L 222.6625 123.1258 \n",
       "L 219.610938 123.14749 \n",
       "L 216.559375 123.162982 \n",
       "L 213.507813 123.172278 \n",
       "L 210.45625 123.175376 \n",
       "L 207.404688 123.172278 \n",
       "L 204.353125 123.162982 \n",
       "L 201.301563 123.14749 \n",
       "L 198.25 123.1258 \n",
       "L 195.198438 123.097914 \n",
       "L 192.146875 123.06383 \n",
       "L 189.095313 123.02355 \n",
       "L 186.04375 122.977073 \n",
       "L 182.992188 122.924398 \n",
       "L 179.940625 122.865527 \n",
       "L 176.889063 122.800458 \n",
       "L 173.8375 122.729193 \n",
       "L 170.785938 122.651731 \n",
       "L 169.542708 122.617647 \n",
       "L 167.734375 122.565653 \n",
       "L 164.682813 122.471413 \n",
       "L 161.63125 122.370674 \n",
       "L 158.579688 122.263436 \n",
       "L 155.528125 122.149699 \n",
       "L 152.476563 122.029462 \n",
       "L 149.425 121.902726 \n",
       "L 146.373438 121.769491 \n",
       "L 143.321875 121.629756 \n",
       "L 140.270313 121.483522 \n",
       "L 137.21875 121.330789 \n",
       "L 134.167188 121.171557 \n",
       "L 131.115625 121.005825 \n",
       "L 128.064063 120.833594 \n",
       "L 125.0125 120.654864 \n",
       "L 121.960938 120.469634 \n",
       "L 118.909375 120.277905 \n",
       "L 115.857813 120.079677 \n",
       "L 113.96875 119.952941 \n",
       "L 112.80625 119.87095 \n",
       "L 109.754688 119.648891 \n",
       "L 106.703125 119.42 \n",
       "L 103.651563 119.184276 \n",
       "L 100.6 118.941719 \n",
       "L 97.548438 118.69233 \n",
       "L 94.496875 118.436109 \n",
       "L 91.445313 118.173054 \n",
       "L 88.39375 117.903167 \n",
       "L 85.342188 117.626448 \n",
       "L 82.290625 117.342896 \n",
       "L 81.716213 117.288235 \n",
       "L 79.239063 117.039769 \n",
       "L 76.1875 116.726486 \n",
       "L 73.135938 116.406002 \n",
       "L 70.084375 116.078315 \n",
       "L 67.032813 115.743426 \n",
       "L 63.98125 115.401335 \n",
       "L 60.929688 115.052043 \n",
       "L 57.878125 114.695548 \n",
       "L 57.273855 114.623529 \n",
       "L 54.826563 114.315185 \n",
       "L 51.775 113.923092 \n",
       "L 48.723438 113.523387 \n",
       "L 45.671875 113.116067 \n",
       "L 42.620313 112.701134 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pd103e112b8)\" d=\"M 42.620313 20.616513 \n",
       "L 45.671875 20.20158 \n",
       "L 48.723438 19.794261 \n",
       "L 51.775 19.394555 \n",
       "L 54.826563 19.002462 \n",
       "L 57.273855 18.694118 \n",
       "L 57.878125 18.622099 \n",
       "L 60.929688 18.265604 \n",
       "L 63.98125 17.916312 \n",
       "L 67.032813 17.574221 \n",
       "L 70.084375 17.239332 \n",
       "L 73.135938 16.911645 \n",
       "L 76.1875 16.591161 \n",
       "L 79.239063 16.277878 \n",
       "L 81.716213 16.029412 \n",
       "L 82.290625 15.974751 \n",
       "L 85.342188 15.691199 \n",
       "L 88.39375 15.41448 \n",
       "L 91.445313 15.144593 \n",
       "L 94.496875 14.881538 \n",
       "L 97.548438 14.625317 \n",
       "L 100.6 14.375928 \n",
       "L 103.651563 14.133371 \n",
       "L 106.703125 13.897647 \n",
       "L 109.754688 13.668756 \n",
       "L 112.80625 13.446697 \n",
       "L 113.96875 13.364706 \n",
       "L 115.857813 13.23797 \n",
       "L 118.909375 13.039742 \n",
       "L 121.960938 12.848013 \n",
       "L 125.0125 12.662783 \n",
       "L 128.064063 12.484053 \n",
       "L 131.115625 12.311822 \n",
       "L 134.167188 12.14609 \n",
       "L 137.21875 11.986858 \n",
       "L 140.270313 11.834125 \n",
       "L 143.321875 11.687891 \n",
       "L 146.373438 11.548156 \n",
       "L 149.425 11.414921 \n",
       "L 152.476563 11.288185 \n",
       "L 155.528125 11.167948 \n",
       "L 158.579688 11.054211 \n",
       "L 161.63125 10.946973 \n",
       "L 164.682813 10.846234 \n",
       "L 167.734375 10.751994 \n",
       "L 169.542708 10.7 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_4\">\n",
       "    <path clip-path=\"url(#pd103e112b8)\" d=\"M 237.920313 131.696681 \n",
       "L 234.86875 131.742905 \n",
       "L 231.817188 131.783691 \n",
       "L 228.765625 131.81904 \n",
       "L 225.714063 131.84895 \n",
       "L 222.6625 131.873421 \n",
       "L 219.610938 131.892455 \n",
       "L 216.559375 131.90605 \n",
       "L 213.507813 131.914208 \n",
       "L 210.45625 131.916927 \n",
       "L 207.404688 131.914208 \n",
       "L 204.353125 131.90605 \n",
       "L 201.301563 131.892455 \n",
       "L 198.25 131.873421 \n",
       "L 195.198438 131.84895 \n",
       "L 192.146875 131.81904 \n",
       "L 189.095313 131.783691 \n",
       "L 186.04375 131.742905 \n",
       "L 182.992188 131.696681 \n",
       "L 179.940625 131.645018 \n",
       "L 176.889063 131.587917 \n",
       "L 173.8375 131.525378 \n",
       "L 170.785938 131.457401 \n",
       "L 167.734375 131.383986 \n",
       "L 164.682813 131.305132 \n",
       "L 161.63125 131.22084 \n",
       "L 158.579688 131.13111 \n",
       "L 155.528125 131.035942 \n",
       "L 152.476563 130.935336 \n",
       "L 149.425 130.829292 \n",
       "L 146.373438 130.717809 \n",
       "L 143.605741 130.611765 \n",
       "L 143.321875 130.600426 \n",
       "L 140.270313 130.47286 \n",
       "L 137.21875 130.339625 \n",
       "L 134.167188 130.20072 \n",
       "L 131.115625 130.056145 \n",
       "L 128.064063 129.905901 \n",
       "L 125.0125 129.749987 \n",
       "L 121.960938 129.588404 \n",
       "L 118.909375 129.421151 \n",
       "L 115.857813 129.248229 \n",
       "L 112.80625 129.069637 \n",
       "L 109.754688 128.885375 \n",
       "L 106.703125 128.695444 \n",
       "L 103.651563 128.499844 \n",
       "L 100.6 128.298573 \n",
       "L 97.548438 128.091633 \n",
       "L 95.473375 127.947059 \n",
       "L 94.496875 127.876 \n",
       "L 91.445313 127.64802 \n",
       "L 88.39375 127.414118 \n",
       "L 85.342188 127.174294 \n",
       "L 82.290625 126.928549 \n",
       "L 79.239063 126.676882 \n",
       "L 76.1875 126.419294 \n",
       "L 73.135938 126.155784 \n",
       "L 70.084375 125.886353 \n",
       "L 67.032813 125.611 \n",
       "L 63.98125 125.329725 \n",
       "L 63.477899 125.282353 \n",
       "L 60.929688 125.031375 \n",
       "L 57.878125 124.724624 \n",
       "L 54.826563 124.411676 \n",
       "L 51.775 124.092531 \n",
       "L 48.723438 123.767189 \n",
       "L 45.671875 123.43565 \n",
       "L 42.620313 123.097914 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_5\">\n",
       "    <path clip-path=\"url(#pd103e112b8)\" d=\"M 237.920313 139.427096 \n",
       "L 234.86875 139.468278 \n",
       "L 231.817188 139.504615 \n",
       "L 228.765625 139.536107 \n",
       "L 225.714063 139.562754 \n",
       "L 222.6625 139.584556 \n",
       "L 219.610938 139.601513 \n",
       "L 216.559375 139.613626 \n",
       "L 213.507813 139.620893 \n",
       "L 210.45625 139.623316 \n",
       "L 207.404688 139.620893 \n",
       "L 204.353125 139.613626 \n",
       "L 201.301563 139.601513 \n",
       "L 198.25 139.584556 \n",
       "L 195.198438 139.562754 \n",
       "L 192.146875 139.536107 \n",
       "L 189.095313 139.504615 \n",
       "L 186.04375 139.468278 \n",
       "L 182.992188 139.427096 \n",
       "L 179.940625 139.38107 \n",
       "L 176.889063 139.330198 \n",
       "L 173.8375 139.274481 \n",
       "L 170.785938 139.21392 \n",
       "L 167.734375 139.148513 \n",
       "L 164.682813 139.078262 \n",
       "L 161.63125 139.003166 \n",
       "L 158.579688 138.923225 \n",
       "L 155.528125 138.838439 \n",
       "L 152.476563 138.748807 \n",
       "L 149.425 138.654332 \n",
       "L 147.936433 138.605882 \n",
       "L 146.373438 138.553091 \n",
       "L 143.321875 138.444994 \n",
       "L 140.270313 138.33187 \n",
       "L 137.21875 138.213718 \n",
       "L 134.167188 138.090538 \n",
       "L 131.115625 137.962331 \n",
       "L 128.064063 137.829095 \n",
       "L 125.0125 137.690832 \n",
       "L 121.960938 137.547542 \n",
       "L 118.909375 137.399223 \n",
       "L 115.857813 137.245877 \n",
       "L 112.80625 137.087503 \n",
       "L 109.754688 136.924101 \n",
       "L 106.703125 136.755671 \n",
       "L 103.651563 136.582214 \n",
       "L 100.6 136.403729 \n",
       "L 97.548438 136.220216 \n",
       "L 94.496875 136.031676 \n",
       "L 93.07017 135.941176 \n",
       "L 91.445313 135.834066 \n",
       "L 88.39375 135.627682 \n",
       "L 85.342188 135.416073 \n",
       "L 82.290625 135.199239 \n",
       "L 79.239063 134.97718 \n",
       "L 76.1875 134.749896 \n",
       "L 73.135938 134.517388 \n",
       "L 70.084375 134.279654 \n",
       "L 67.032813 134.036696 \n",
       "L 63.98125 133.788512 \n",
       "L 60.929688 133.535104 \n",
       "L 57.878125 133.276471 \n",
       "L 57.878125 133.276471 \n",
       "L 54.826563 133.001843 \n",
       "L 51.775 132.721777 \n",
       "L 48.723438 132.436273 \n",
       "L 45.671875 132.14533 \n",
       "L 42.620313 131.84895 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_6\">\n",
       "    <path clip-path=\"url(#pd103e112b8)\" d=\"M 210.45625 146.6 \n",
       "L 207.404688 146.597742 \n",
       "L 204.353125 146.590967 \n",
       "L 201.301563 146.579676 \n",
       "L 198.25 146.563868 \n",
       "L 195.198438 146.543544 \n",
       "L 192.146875 146.518704 \n",
       "L 189.095313 146.489347 \n",
       "L 186.04375 146.455474 \n",
       "L 182.992188 146.417084 \n",
       "L 179.940625 146.374177 \n",
       "L 176.889063 146.326755 \n",
       "L 173.8375 146.274816 \n",
       "L 170.785938 146.21836 \n",
       "L 167.734375 146.157388 \n",
       "L 164.682813 146.091899 \n",
       "L 161.63125 146.021894 \n",
       "L 158.579688 145.947373 \n",
       "L 155.528125 145.868335 \n",
       "L 152.476563 145.784781 \n",
       "L 149.425 145.69671 \n",
       "L 146.373438 145.604123 \n",
       "L 143.321875 145.507019 \n",
       "L 140.270313 145.405399 \n",
       "L 137.21875 145.299262 \n",
       "L 134.167188 145.188609 \n",
       "L 131.115625 145.07344 \n",
       "L 128.064063 144.953754 \n",
       "L 125.0125 144.829551 \n",
       "L 121.960938 144.700833 \n",
       "L 118.909375 144.567597 \n",
       "L 115.857813 144.429845 \n",
       "L 112.80625 144.287577 \n",
       "L 109.754688 144.140793 \n",
       "L 106.703125 143.989492 \n",
       "L 105.641712 143.935294 \n",
       "L 103.651563 143.830108 \n",
       "L 100.6 143.664149 \n",
       "L 97.548438 143.493514 \n",
       "L 94.496875 143.318204 \n",
       "L 91.445313 143.13822 \n",
       "L 88.39375 142.95356 \n",
       "L 85.342188 142.764226 \n",
       "L 82.290625 142.570217 \n",
       "L 79.239063 142.371533 \n",
       "L 76.1875 142.168173 \n",
       "L 73.135938 141.960139 \n",
       "L 70.084375 141.74743 \n",
       "L 67.032813 141.530046 \n",
       "L 63.98125 141.307988 \n",
       "L 63.477899 141.270588 \n",
       "L 60.929688 141.074369 \n",
       "L 57.878125 140.834545 \n",
       "L 54.826563 140.589877 \n",
       "L 51.775 140.340364 \n",
       "L 48.723438 140.086005 \n",
       "L 45.671875 139.826802 \n",
       "L 42.620313 139.562754 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pd103e112b8)\" d=\"M 237.920313 146.417084 \n",
       "L 234.86875 146.455474 \n",
       "L 231.817188 146.489347 \n",
       "L 228.765625 146.518704 \n",
       "L 225.714063 146.543544 \n",
       "L 222.6625 146.563868 \n",
       "L 219.610938 146.579676 \n",
       "L 216.559375 146.590967 \n",
       "L 213.507813 146.597742 \n",
       "L 210.45625 146.6 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_7\">\n",
       "    <path clip-path=\"url(#pd103e112b8)\" d=\"M 43.320212 146.6 \n",
       "L 42.620313 146.543544 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_10\">\n",
       "    <path clip-path=\"url(#pd103e112b8)\" d=\"M 57.878125 119.952941 \n",
       "L 70.084375 34.682353 \n",
       "L 81.314125 85.844706 \n",
       "L 91.645495 55.147294 \n",
       "L 101.150355 73.565741 \n",
       "L 109.894827 62.514673 \n",
       "L 117.939741 69.145314 \n",
       "L 125.341062 65.166929 \n",
       "L 132.150277 67.55396 \n",
       "L 138.414754 66.121742 \n",
       "L 144.178074 66.981073 \n",
       "L 149.480328 66.465474 \n",
       "L 154.358402 66.774833 \n",
       "L 158.84623 66.589218 \n",
       "L 162.975031 66.700587 \n",
       "L 166.773529 66.633765 \n",
       "L 170.268147 66.673858 \n",
       "L 173.483195 66.649803 \n",
       "L 176.441039 66.664236 \n",
       "L 179.162256 66.655576 \n",
       "L 181.665776 66.660772 \n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    <defs>\n",
       "     <path d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" id=\"mfca07fe722\" style=\"stroke:#ff7f0e;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#pd103e112b8)\">\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"57.878125\" xlink:href=\"#mfca07fe722\" y=\"119.952941\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"70.084375\" xlink:href=\"#mfca07fe722\" y=\"34.682353\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"81.314125\" xlink:href=\"#mfca07fe722\" y=\"85.844706\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"91.645495\" xlink:href=\"#mfca07fe722\" y=\"55.147294\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"101.150355\" xlink:href=\"#mfca07fe722\" y=\"73.565741\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"109.894827\" xlink:href=\"#mfca07fe722\" y=\"62.514673\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"117.939741\" xlink:href=\"#mfca07fe722\" y=\"69.145314\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"125.341062\" xlink:href=\"#mfca07fe722\" y=\"65.166929\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"132.150277\" xlink:href=\"#mfca07fe722\" y=\"67.55396\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"138.414754\" xlink:href=\"#mfca07fe722\" y=\"66.121742\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"144.178074\" xlink:href=\"#mfca07fe722\" y=\"66.981073\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"149.480328\" xlink:href=\"#mfca07fe722\" y=\"66.465474\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"154.358402\" xlink:href=\"#mfca07fe722\" y=\"66.774833\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"158.84623\" xlink:href=\"#mfca07fe722\" y=\"66.589218\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"162.975031\" xlink:href=\"#mfca07fe722\" y=\"66.700587\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"166.773529\" xlink:href=\"#mfca07fe722\" y=\"66.633765\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"170.268147\" xlink:href=\"#mfca07fe722\" y=\"66.673858\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"173.483195\" xlink:href=\"#mfca07fe722\" y=\"66.649803\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"176.441039\" xlink:href=\"#mfca07fe722\" y=\"66.664236\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"179.162256\" xlink:href=\"#mfca07fe722\" y=\"66.655576\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"181.665776\" xlink:href=\"#mfca07fe722\" y=\"66.660772\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 42.620312 146.6 \n",
       "L 42.620312 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 237.920313 146.6 \n",
       "L 237.920313 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 42.620313 146.6 \n",
       "L 237.920313 146.6 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 42.620313 10.7 \n",
       "L 237.920313 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pd103e112b8\">\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"42.620312\" y=\"10.7\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import d2ltorch as d2lt\n",
    "import torch\n",
    "from torch import optim\n",
    "\n",
    "eta = 0.4\n",
    "\n",
    "def f_2d(x1, x2):\n",
    "    return 0.1 * x1 ** 2 + 2 * x2 ** 2\n",
    "\n",
    "def gd_2d(x1, x2, s1, s2):\n",
    "    return (x1 - eta * 0.2 * x1, x2 - eta * 4 * x2, 0, 0)\n",
    "\n",
    "d2lt.show_trace_2d(f_2d, d2lt.train_2d(gd_2d))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，同一位置上，目标函数在竖直方向（$x_2$轴方向）比在水平方向（$x_1$轴方向）的斜率的绝对值更大。因此，给定学习率，梯度下降迭代自变量时会使自变量在竖直方向比在水平方向移动幅度更大。那么，我们需要一个较小的学习率从而避免自变量在竖直方向上越过目标函数最优解。然而，这会造成自变量在水平方向上朝最优解移动变慢。\n",
    "\n",
    "下面我们试着将学习率调得稍大一点，此时自变量在竖直方向不断越过最优解并逐渐发散。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "4"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 20, x1 -0.387814, x2 -1673.365109\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 267.707812 184.15625\" width=\"267.707812pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 184.15625 \n",
       "L 267.707812 184.15625 \n",
       "L 267.707812 -0 \n",
       "L 0 -0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 61.707813 146.6 \n",
       "L 257.007812 146.6 \n",
       "L 257.007812 10.7 \n",
       "L 61.707813 10.7 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"m26799ad06a\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"107.48125\" xlink:href=\"#m26799ad06a\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −4 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-2212\"/>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"DejaVuSans-34\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(100.110156 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"168.5125\" xlink:href=\"#m26799ad06a\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −2 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-32\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(161.141406 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"229.54375\" xlink:href=\"#m26799ad06a\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-30\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(226.3625 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_4\">\n",
       "     <!-- x1 -->\n",
       "     <defs>\n",
       "      <path d=\"M 54.890625 54.6875 \n",
       "L 35.109375 28.078125 \n",
       "L 55.90625 0 \n",
       "L 45.3125 0 \n",
       "L 29.390625 21.484375 \n",
       "L 13.484375 0 \n",
       "L 2.875 0 \n",
       "L 24.125 28.609375 \n",
       "L 4.6875 54.6875 \n",
       "L 15.28125 54.6875 \n",
       "L 29.78125 35.203125 \n",
       "L 44.28125 54.6875 \n",
       "z\n",
       "\" id=\"DejaVuSans-78\"/>\n",
       "      <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-31\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(153.217188 174.876562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "      <use x=\"59.179688\" xlink:href=\"#DejaVuSans-31\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"m0684262908\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"61.707813\" xlink:href=\"#m0684262908\" y=\"138.345204\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- −1500 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"DejaVuSans-35\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(20.878125 142.144422)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use x=\"211.035156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"274.658203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"61.707813\" xlink:href=\"#m0684262908\" y=\"114.674194\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- −1000 -->\n",
       "      <g transform=\"translate(20.878125 118.473413)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"211.035156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"274.658203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"61.707813\" xlink:href=\"#m0684262908\" y=\"91.003185\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- −500 -->\n",
       "      <g transform=\"translate(27.240625 94.802403)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use x=\"147.412109\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"211.035156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"61.707813\" xlink:href=\"#m0684262908\" y=\"67.332175\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(48.345313 71.131394)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"61.707813\" xlink:href=\"#m0684262908\" y=\"43.661166\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 500 -->\n",
       "      <g transform=\"translate(35.620312 47.460384)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"61.707813\" xlink:href=\"#m0684262908\" y=\"19.990156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 1000 -->\n",
       "      <g transform=\"translate(29.257812 23.789375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"190.869141\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- x2 -->\n",
       "     <g transform=\"translate(14.798438 84.790625)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "      <use x=\"59.179688\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_1\">\n",
       "    <path clip-path=\"url(#pb767c309e6)\" d=\"M 257.007812 97.273895 \n",
       "L 253.95625 97.273896 \n",
       "L 250.904688 97.273896 \n",
       "L 247.853125 97.273896 \n",
       "L 244.801563 97.273896 \n",
       "L 241.75 97.273897 \n",
       "L 238.698438 97.273897 \n",
       "L 235.646875 97.273897 \n",
       "L 232.595313 97.273897 \n",
       "L 229.54375 97.273897 \n",
       "L 226.492188 97.273897 \n",
       "L 223.440625 97.273897 \n",
       "L 220.389063 97.273897 \n",
       "L 217.3375 97.273897 \n",
       "L 214.285938 97.273896 \n",
       "L 211.234375 97.273896 \n",
       "L 208.182813 97.273896 \n",
       "L 205.13125 97.273896 \n",
       "L 202.079688 97.273895 \n",
       "L 199.028125 97.273895 \n",
       "L 195.976562 97.273895 \n",
       "L 192.925 97.273894 \n",
       "L 189.873438 97.273894 \n",
       "L 186.821875 97.273893 \n",
       "L 183.770313 97.273893 \n",
       "L 180.71875 97.273892 \n",
       "L 177.667188 97.273891 \n",
       "L 174.615625 97.273891 \n",
       "L 171.564063 97.27389 \n",
       "L 168.5125 97.273889 \n",
       "L 165.460938 97.273889 \n",
       "L 162.409375 97.273888 \n",
       "L 159.357813 97.273887 \n",
       "L 156.30625 97.273886 \n",
       "L 153.254688 97.273885 \n",
       "L 150.203125 97.273884 \n",
       "L 147.151563 97.273883 \n",
       "L 144.1 97.273882 \n",
       "L 141.048438 97.273881 \n",
       "L 137.996875 97.27388 \n",
       "L 134.945312 97.273879 \n",
       "L 131.89375 97.273878 \n",
       "L 128.842188 97.273877 \n",
       "L 125.790625 97.273875 \n",
       "L 122.739063 97.273874 \n",
       "L 119.6875 97.273873 \n",
       "L 116.635938 97.273871 \n",
       "L 113.584375 97.27387 \n",
       "L 110.532813 97.273868 \n",
       "L 107.48125 97.273867 \n",
       "L 104.429688 97.273865 \n",
       "L 101.378125 97.273864 \n",
       "L 98.326563 97.273862 \n",
       "L 95.275 97.273861 \n",
       "L 92.223438 97.273859 \n",
       "L 89.171875 97.273857 \n",
       "L 86.120313 97.273856 \n",
       "L 83.06875 97.273854 \n",
       "L 80.017188 97.273852 \n",
       "L 76.965625 97.27385 \n",
       "L 73.914062 97.273848 \n",
       "L 70.8625 97.273846 \n",
       "L 67.810937 97.273844 \n",
       "L 64.759375 97.273842 \n",
       "L 61.707813 97.27384 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pb767c309e6)\" d=\"M 61.707813 37.39051 \n",
       "L 64.759375 37.390508 \n",
       "L 67.810937 37.390506 \n",
       "L 70.8625 37.390504 \n",
       "L 73.914062 37.390502 \n",
       "L 76.965625 37.3905 \n",
       "L 80.017188 37.390498 \n",
       "L 83.06875 37.390496 \n",
       "L 86.120313 37.390495 \n",
       "L 89.171875 37.390493 \n",
       "L 92.223438 37.390491 \n",
       "L 95.275 37.390489 \n",
       "L 98.326563 37.390488 \n",
       "L 101.378125 37.390486 \n",
       "L 104.429688 37.390485 \n",
       "L 107.48125 37.390483 \n",
       "L 110.532813 37.390482 \n",
       "L 113.584375 37.39048 \n",
       "L 116.635938 37.390479 \n",
       "L 119.6875 37.390478 \n",
       "L 122.739063 37.390476 \n",
       "L 125.790625 37.390475 \n",
       "L 128.842188 37.390474 \n",
       "L 131.89375 37.390472 \n",
       "L 134.945312 37.390471 \n",
       "L 137.996875 37.39047 \n",
       "L 141.048438 37.390469 \n",
       "L 144.1 37.390468 \n",
       "L 147.151563 37.390467 \n",
       "L 150.203125 37.390466 \n",
       "L 153.254688 37.390465 \n",
       "L 156.30625 37.390464 \n",
       "L 159.357813 37.390463 \n",
       "L 162.409375 37.390462 \n",
       "L 165.460938 37.390462 \n",
       "L 168.5125 37.390461 \n",
       "L 171.564063 37.39046 \n",
       "L 174.615625 37.390459 \n",
       "L 177.667188 37.390459 \n",
       "L 180.71875 37.390458 \n",
       "L 183.770313 37.390457 \n",
       "L 186.821875 37.390457 \n",
       "L 189.873438 37.390456 \n",
       "L 192.925 37.390456 \n",
       "L 195.976562 37.390456 \n",
       "L 199.028125 37.390455 \n",
       "L 202.079688 37.390455 \n",
       "L 205.13125 37.390454 \n",
       "L 208.182813 37.390454 \n",
       "L 211.234375 37.390454 \n",
       "L 214.285938 37.390454 \n",
       "L 217.3375 37.390454 \n",
       "L 220.389063 37.390453 \n",
       "L 223.440625 37.390453 \n",
       "L 226.492188 37.390453 \n",
       "L 229.54375 37.390453 \n",
       "L 232.595313 37.390453 \n",
       "L 235.646875 37.390453 \n",
       "L 238.698438 37.390453 \n",
       "L 241.75 37.390454 \n",
       "L 244.801563 37.390454 \n",
       "L 247.853125 37.390454 \n",
       "L 250.904688 37.390454 \n",
       "L 253.95625 37.390454 \n",
       "L 257.007812 37.390455 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_2\">\n",
       "    <path clip-path=\"url(#pb767c309e6)\" d=\"M 257.007812 109.676163 \n",
       "L 253.95625 109.676163 \n",
       "L 250.904688 109.676164 \n",
       "L 247.853125 109.676164 \n",
       "L 244.801563 109.676164 \n",
       "L 241.75 109.676164 \n",
       "L 238.698438 109.676164 \n",
       "L 235.646875 109.676164 \n",
       "L 232.595313 109.676164 \n",
       "L 229.54375 109.676164 \n",
       "L 226.492188 109.676164 \n",
       "L 223.440625 109.676164 \n",
       "L 220.389063 109.676164 \n",
       "L 217.3375 109.676164 \n",
       "L 214.285938 109.676164 \n",
       "L 211.234375 109.676164 \n",
       "L 208.182813 109.676164 \n",
       "L 205.13125 109.676163 \n",
       "L 202.079688 109.676163 \n",
       "L 199.028125 109.676163 \n",
       "L 195.976562 109.676163 \n",
       "L 192.925 109.676162 \n",
       "L 189.873438 109.676162 \n",
       "L 186.821875 109.676162 \n",
       "L 183.770313 109.676161 \n",
       "L 180.71875 109.676161 \n",
       "L 177.667188 109.67616 \n",
       "L 174.615625 109.67616 \n",
       "L 171.564063 109.676159 \n",
       "L 168.5125 109.676159 \n",
       "L 165.460938 109.676158 \n",
       "L 162.409375 109.676158 \n",
       "L 159.357813 109.676157 \n",
       "L 156.30625 109.676157 \n",
       "L 153.254688 109.676156 \n",
       "L 150.203125 109.676155 \n",
       "L 147.151563 109.676155 \n",
       "L 144.1 109.676154 \n",
       "L 141.048438 109.676153 \n",
       "L 137.996875 109.676152 \n",
       "L 134.945312 109.676151 \n",
       "L 131.89375 109.676151 \n",
       "L 128.842188 109.67615 \n",
       "L 125.790625 109.676149 \n",
       "L 122.739063 109.676148 \n",
       "L 119.6875 109.676147 \n",
       "L 116.635938 109.676146 \n",
       "L 113.584375 109.676145 \n",
       "L 110.532813 109.676144 \n",
       "L 107.48125 109.676143 \n",
       "L 104.429688 109.676142 \n",
       "L 101.378125 109.676141 \n",
       "L 98.326563 109.67614 \n",
       "L 95.275 109.676139 \n",
       "L 92.223438 109.676137 \n",
       "L 89.171875 109.676136 \n",
       "L 86.120313 109.676135 \n",
       "L 83.06875 109.676134 \n",
       "L 80.017188 109.676132 \n",
       "L 76.965625 109.676131 \n",
       "L 73.914062 109.67613 \n",
       "L 70.8625 109.676128 \n",
       "L 67.810937 109.676127 \n",
       "L 64.759375 109.676126 \n",
       "L 61.707813 109.676124 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pb767c309e6)\" d=\"M 61.707813 24.988226 \n",
       "L 64.759375 24.988225 \n",
       "L 67.810938 24.988223 \n",
       "L 70.8625 24.988222 \n",
       "L 73.914062 24.98822 \n",
       "L 76.965625 24.988219 \n",
       "L 80.017188 24.988218 \n",
       "L 83.06875 24.988216 \n",
       "L 86.120313 24.988215 \n",
       "L 89.171875 24.988214 \n",
       "L 92.223438 24.988213 \n",
       "L 95.275 24.988212 \n",
       "L 98.326563 24.98821 \n",
       "L 101.378125 24.988209 \n",
       "L 104.429688 24.988208 \n",
       "L 107.48125 24.988207 \n",
       "L 110.532813 24.988206 \n",
       "L 113.584375 24.988205 \n",
       "L 116.635938 24.988204 \n",
       "L 119.6875 24.988203 \n",
       "L 122.739063 24.988202 \n",
       "L 125.790625 24.988201 \n",
       "L 128.842188 24.9882 \n",
       "L 131.89375 24.988199 \n",
       "L 134.945312 24.988199 \n",
       "L 137.996875 24.988198 \n",
       "L 141.048438 24.988197 \n",
       "L 144.1 24.988196 \n",
       "L 147.151563 24.988196 \n",
       "L 150.203125 24.988195 \n",
       "L 153.254688 24.988194 \n",
       "L 156.30625 24.988194 \n",
       "L 159.357813 24.988193 \n",
       "L 162.409375 24.988192 \n",
       "L 165.460938 24.988192 \n",
       "L 168.5125 24.988191 \n",
       "L 171.564063 24.988191 \n",
       "L 174.615625 24.98819 \n",
       "L 177.667188 24.98819 \n",
       "L 180.71875 24.988189 \n",
       "L 183.770313 24.988189 \n",
       "L 186.821875 24.988189 \n",
       "L 189.873438 24.988188 \n",
       "L 192.925 24.988188 \n",
       "L 195.976562 24.988188 \n",
       "L 199.028125 24.988187 \n",
       "L 202.079688 24.988187 \n",
       "L 205.13125 24.988187 \n",
       "L 208.182813 24.988187 \n",
       "L 211.234375 24.988186 \n",
       "L 214.285938 24.988186 \n",
       "L 217.3375 24.988186 \n",
       "L 220.389063 24.988186 \n",
       "L 223.440625 24.988186 \n",
       "L 226.492188 24.988186 \n",
       "L 229.54375 24.988186 \n",
       "L 232.595313 24.988186 \n",
       "L 235.646875 24.988186 \n",
       "L 238.698438 24.988186 \n",
       "L 241.75 24.988186 \n",
       "L 244.801563 24.988186 \n",
       "L 247.853125 24.988186 \n",
       "L 250.904688 24.988187 \n",
       "L 253.95625 24.988187 \n",
       "L 257.007812 24.988187 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_3\">\n",
       "    <path clip-path=\"url(#pb767c309e6)\" d=\"M 257.007812 119.192758 \n",
       "L 253.95625 119.192758 \n",
       "L 250.904688 119.192758 \n",
       "L 247.853125 119.192758 \n",
       "L 244.801563 119.192758 \n",
       "L 241.75 119.192758 \n",
       "L 238.698438 119.192758 \n",
       "L 235.646875 119.192759 \n",
       "L 232.595313 119.192759 \n",
       "L 229.54375 119.192759 \n",
       "L 226.492188 119.192759 \n",
       "L 223.440625 119.192759 \n",
       "L 220.389063 119.192758 \n",
       "L 217.3375 119.192758 \n",
       "L 214.285938 119.192758 \n",
       "L 211.234375 119.192758 \n",
       "L 208.182813 119.192758 \n",
       "L 205.13125 119.192758 \n",
       "L 202.079688 119.192758 \n",
       "L 199.028125 119.192758 \n",
       "L 195.976562 119.192757 \n",
       "L 192.925 119.192757 \n",
       "L 189.873438 119.192757 \n",
       "L 186.821875 119.192756 \n",
       "L 183.770313 119.192756 \n",
       "L 180.71875 119.192756 \n",
       "L 177.667188 119.192755 \n",
       "L 174.615625 119.192755 \n",
       "L 171.564063 119.192755 \n",
       "L 168.5125 119.192754 \n",
       "L 165.460938 119.192754 \n",
       "L 162.409375 119.192753 \n",
       "L 159.357813 119.192753 \n",
       "L 156.30625 119.192752 \n",
       "L 153.254688 119.192752 \n",
       "L 150.203125 119.192751 \n",
       "L 147.151563 119.192751 \n",
       "L 144.1 119.19275 \n",
       "L 141.048438 119.192749 \n",
       "L 137.996875 119.192749 \n",
       "L 134.945312 119.192748 \n",
       "L 131.89375 119.192748 \n",
       "L 128.842188 119.192747 \n",
       "L 125.790625 119.192746 \n",
       "L 122.739063 119.192745 \n",
       "L 119.6875 119.192745 \n",
       "L 116.635938 119.192744 \n",
       "L 113.584375 119.192743 \n",
       "L 110.532813 119.192742 \n",
       "L 107.48125 119.192741 \n",
       "L 104.429688 119.19274 \n",
       "L 101.378125 119.19274 \n",
       "L 98.326563 119.192739 \n",
       "L 95.275 119.192738 \n",
       "L 92.223438 119.192737 \n",
       "L 89.171875 119.192736 \n",
       "L 86.120313 119.192735 \n",
       "L 83.06875 119.192734 \n",
       "L 80.017188 119.192733 \n",
       "L 76.965625 119.192732 \n",
       "L 73.914062 119.19273 \n",
       "L 70.8625 119.192729 \n",
       "L 67.810937 119.192728 \n",
       "L 64.759375 119.192727 \n",
       "L 61.707813 119.192726 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pb767c309e6)\" d=\"M 61.707813 15.471624 \n",
       "L 64.759375 15.471623 \n",
       "L 67.810937 15.471622 \n",
       "L 70.8625 15.471621 \n",
       "L 73.914062 15.47162 \n",
       "L 76.965625 15.471619 \n",
       "L 80.017188 15.471617 \n",
       "L 83.06875 15.471616 \n",
       "L 86.120313 15.471615 \n",
       "L 89.171875 15.471614 \n",
       "L 92.223438 15.471613 \n",
       "L 95.275 15.471612 \n",
       "L 98.326563 15.471612 \n",
       "L 101.378125 15.471611 \n",
       "L 104.429688 15.47161 \n",
       "L 107.48125 15.471609 \n",
       "L 110.532813 15.471608 \n",
       "L 113.584375 15.471607 \n",
       "L 116.635938 15.471606 \n",
       "L 119.6875 15.471606 \n",
       "L 122.739063 15.471605 \n",
       "L 125.790625 15.471604 \n",
       "L 128.842188 15.471603 \n",
       "L 131.89375 15.471603 \n",
       "L 134.945312 15.471602 \n",
       "L 137.996875 15.471601 \n",
       "L 141.048438 15.471601 \n",
       "L 144.1 15.4716 \n",
       "L 147.151563 15.471599 \n",
       "L 150.203125 15.471599 \n",
       "L 153.254688 15.471598 \n",
       "L 156.30625 15.471598 \n",
       "L 159.357813 15.471597 \n",
       "L 162.409375 15.471597 \n",
       "L 165.460938 15.471596 \n",
       "L 168.5125 15.471596 \n",
       "L 171.564063 15.471595 \n",
       "L 174.615625 15.471595 \n",
       "L 177.667188 15.471595 \n",
       "L 180.71875 15.471594 \n",
       "L 183.770313 15.471594 \n",
       "L 186.821875 15.471594 \n",
       "L 189.873438 15.471593 \n",
       "L 192.925 15.471593 \n",
       "L 195.976562 15.471593 \n",
       "L 199.028125 15.471593 \n",
       "L 202.079688 15.471592 \n",
       "L 205.13125 15.471592 \n",
       "L 208.182813 15.471592 \n",
       "L 211.234375 15.471592 \n",
       "L 214.285938 15.471592 \n",
       "L 217.3375 15.471592 \n",
       "L 220.389063 15.471592 \n",
       "L 223.440625 15.471592 \n",
       "L 226.492188 15.471592 \n",
       "L 229.54375 15.471592 \n",
       "L 232.595313 15.471592 \n",
       "L 235.646875 15.471592 \n",
       "L 238.698438 15.471592 \n",
       "L 241.75 15.471592 \n",
       "L 244.801563 15.471592 \n",
       "L 247.853125 15.471592 \n",
       "L 250.904688 15.471592 \n",
       "L 253.95625 15.471592 \n",
       "L 257.007812 15.471592 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_4\">\n",
       "    <path clip-path=\"url(#pb767c309e6)\" d=\"M 257.007812 127.215618 \n",
       "L 253.95625 127.215618 \n",
       "L 250.904688 127.215618 \n",
       "L 247.853125 127.215618 \n",
       "L 244.801563 127.215619 \n",
       "L 241.75 127.215619 \n",
       "L 238.698438 127.215619 \n",
       "L 235.646875 127.215619 \n",
       "L 232.595313 127.215619 \n",
       "L 229.54375 127.215619 \n",
       "L 226.492188 127.215619 \n",
       "L 223.440625 127.215619 \n",
       "L 220.389063 127.215619 \n",
       "L 217.3375 127.215619 \n",
       "L 214.285938 127.215619 \n",
       "L 211.234375 127.215618 \n",
       "L 208.182813 127.215618 \n",
       "L 205.13125 127.215618 \n",
       "L 202.079688 127.215618 \n",
       "L 199.028125 127.215618 \n",
       "L 195.976562 127.215618 \n",
       "L 192.925 127.215617 \n",
       "L 189.873438 127.215617 \n",
       "L 186.821875 127.215617 \n",
       "L 183.770313 127.215617 \n",
       "L 180.71875 127.215616 \n",
       "L 177.667188 127.215616 \n",
       "L 174.615625 127.215616 \n",
       "L 171.564063 127.215615 \n",
       "L 168.5125 127.215615 \n",
       "L 165.460938 127.215615 \n",
       "L 162.409375 127.215614 \n",
       "L 159.357813 127.215614 \n",
       "L 156.30625 127.215613 \n",
       "L 153.254688 127.215613 \n",
       "L 150.203125 127.215612 \n",
       "L 147.151563 127.215612 \n",
       "L 144.1 127.215611 \n",
       "L 141.048438 127.215611 \n",
       "L 137.996875 127.21561 \n",
       "L 134.945312 127.21561 \n",
       "L 131.89375 127.215609 \n",
       "L 128.842188 127.215609 \n",
       "L 125.790625 127.215608 \n",
       "L 122.739063 127.215607 \n",
       "L 119.6875 127.215607 \n",
       "L 116.635938 127.215606 \n",
       "L 113.584375 127.215605 \n",
       "L 110.532813 127.215605 \n",
       "L 107.48125 127.215604 \n",
       "L 104.429688 127.215603 \n",
       "L 101.378125 127.215602 \n",
       "L 98.326563 127.215601 \n",
       "L 95.275 127.215601 \n",
       "L 92.223438 127.2156 \n",
       "L 89.171875 127.215599 \n",
       "L 86.120313 127.215598 \n",
       "L 83.06875 127.215597 \n",
       "L 80.017188 127.215596 \n",
       "L 76.965625 127.215595 \n",
       "L 73.914062 127.215594 \n",
       "L 70.8625 127.215593 \n",
       "L 67.810937 127.215592 \n",
       "L 64.759375 127.215591 \n",
       "L 61.707813 127.21559 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_5\">\n",
       "    <path clip-path=\"url(#pb767c309e6)\" d=\"M 257.007812 134.2839 \n",
       "L 253.95625 134.2839 \n",
       "L 250.904688 134.2839 \n",
       "L 247.853125 134.2839 \n",
       "L 244.801563 134.2839 \n",
       "L 241.75 134.2839 \n",
       "L 238.698438 134.2839 \n",
       "L 235.646875 134.2839 \n",
       "L 232.595313 134.2839 \n",
       "L 229.54375 134.2839 \n",
       "L 226.492188 134.2839 \n",
       "L 223.440625 134.2839 \n",
       "L 220.389063 134.2839 \n",
       "L 217.3375 134.2839 \n",
       "L 214.285938 134.2839 \n",
       "L 211.234375 134.2839 \n",
       "L 208.182813 134.2839 \n",
       "L 205.13125 134.2839 \n",
       "L 202.079688 134.2839 \n",
       "L 199.028125 134.2839 \n",
       "L 195.976562 134.283899 \n",
       "L 192.925 134.283899 \n",
       "L 189.873438 134.283899 \n",
       "L 186.821875 134.283899 \n",
       "L 183.770313 134.283899 \n",
       "L 180.71875 134.283898 \n",
       "L 177.667188 134.283898 \n",
       "L 174.615625 134.283898 \n",
       "L 171.564063 134.283897 \n",
       "L 168.5125 134.283897 \n",
       "L 165.460938 134.283897 \n",
       "L 162.409375 134.283896 \n",
       "L 159.357813 134.283896 \n",
       "L 156.30625 134.283896 \n",
       "L 153.254688 134.283895 \n",
       "L 150.203125 134.283895 \n",
       "L 147.151563 134.283894 \n",
       "L 144.1 134.283894 \n",
       "L 141.048438 134.283893 \n",
       "L 137.996875 134.283893 \n",
       "L 134.945312 134.283892 \n",
       "L 131.89375 134.283892 \n",
       "L 128.842188 134.283891 \n",
       "L 125.790625 134.283891 \n",
       "L 122.739063 134.28389 \n",
       "L 119.6875 134.28389 \n",
       "L 116.635938 134.283889 \n",
       "L 113.584375 134.283888 \n",
       "L 110.532813 134.283888 \n",
       "L 107.48125 134.283887 \n",
       "L 104.429688 134.283886 \n",
       "L 101.378125 134.283886 \n",
       "L 98.326563 134.283885 \n",
       "L 95.275 134.283884 \n",
       "L 92.223438 134.283884 \n",
       "L 89.171875 134.283883 \n",
       "L 86.120313 134.283882 \n",
       "L 83.06875 134.283881 \n",
       "L 80.017188 134.28388 \n",
       "L 76.965625 134.28388 \n",
       "L 73.914062 134.283879 \n",
       "L 70.8625 134.283878 \n",
       "L 67.810937 134.283877 \n",
       "L 64.759375 134.283876 \n",
       "L 61.707813 134.283875 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_6\">\n",
       "    <path clip-path=\"url(#pb767c309e6)\" d=\"M 257.007812 140.674115 \n",
       "L 253.95625 140.674115 \n",
       "L 250.904688 140.674115 \n",
       "L 247.853125 140.674115 \n",
       "L 244.801563 140.674115 \n",
       "L 241.75 140.674116 \n",
       "L 238.698438 140.674116 \n",
       "L 235.646875 140.674116 \n",
       "L 232.595313 140.674116 \n",
       "L 229.54375 140.674116 \n",
       "L 226.492188 140.674116 \n",
       "L 223.440625 140.674116 \n",
       "L 220.389063 140.674116 \n",
       "L 217.3375 140.674116 \n",
       "L 214.285938 140.674115 \n",
       "L 211.234375 140.674115 \n",
       "L 208.182813 140.674115 \n",
       "L 205.13125 140.674115 \n",
       "L 202.079688 140.674115 \n",
       "L 199.028125 140.674115 \n",
       "L 195.976562 140.674115 \n",
       "L 192.925 140.674115 \n",
       "L 189.873438 140.674114 \n",
       "L 186.821875 140.674114 \n",
       "L 183.770313 140.674114 \n",
       "L 180.71875 140.674114 \n",
       "L 177.667188 140.674113 \n",
       "L 174.615625 140.674113 \n",
       "L 171.564063 140.674113 \n",
       "L 168.5125 140.674113 \n",
       "L 165.460938 140.674112 \n",
       "L 162.409375 140.674112 \n",
       "L 159.357813 140.674112 \n",
       "L 156.30625 140.674111 \n",
       "L 153.254688 140.674111 \n",
       "L 150.203125 140.67411 \n",
       "L 147.151563 140.67411 \n",
       "L 144.1 140.67411 \n",
       "L 141.048438 140.674109 \n",
       "L 137.996875 140.674109 \n",
       "L 134.945312 140.674108 \n",
       "L 131.89375 140.674108 \n",
       "L 128.842188 140.674107 \n",
       "L 125.790625 140.674107 \n",
       "L 122.739063 140.674106 \n",
       "L 119.6875 140.674106 \n",
       "L 116.635938 140.674105 \n",
       "L 113.584375 140.674105 \n",
       "L 110.532813 140.674104 \n",
       "L 107.48125 140.674103 \n",
       "L 104.429688 140.674103 \n",
       "L 101.378125 140.674102 \n",
       "L 98.326563 140.674102 \n",
       "L 95.275 140.674101 \n",
       "L 92.223438 140.6741 \n",
       "L 89.171875 140.674099 \n",
       "L 86.120313 140.674099 \n",
       "L 83.06875 140.674098 \n",
       "L 80.017188 140.674097 \n",
       "L 76.965625 140.674097 \n",
       "L 73.914062 140.674096 \n",
       "L 70.8625 140.674095 \n",
       "L 67.810937 140.674094 \n",
       "L 64.759375 140.674093 \n",
       "L 61.707813 140.674093 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_7\">\n",
       "    <path clip-path=\"url(#pb767c309e6)\" d=\"M 257.007812 146.550524 \n",
       "L 253.95625 146.550524 \n",
       "L 250.904688 146.550525 \n",
       "L 247.853125 146.550525 \n",
       "L 244.801563 146.550525 \n",
       "L 241.75 146.550525 \n",
       "L 238.698438 146.550525 \n",
       "L 235.646875 146.550525 \n",
       "L 232.595313 146.550525 \n",
       "L 229.54375 146.550525 \n",
       "L 226.492188 146.550525 \n",
       "L 223.440625 146.550525 \n",
       "L 220.389063 146.550525 \n",
       "L 217.3375 146.550525 \n",
       "L 214.285938 146.550525 \n",
       "L 211.234375 146.550525 \n",
       "L 208.182813 146.550525 \n",
       "L 205.13125 146.550524 \n",
       "L 202.079688 146.550524 \n",
       "L 199.028125 146.550524 \n",
       "L 195.976562 146.550524 \n",
       "L 192.925 146.550524 \n",
       "L 189.873438 146.550524 \n",
       "L 186.821875 146.550524 \n",
       "L 183.770313 146.550523 \n",
       "L 180.71875 146.550523 \n",
       "L 177.667188 146.550523 \n",
       "L 174.615625 146.550523 \n",
       "L 171.564063 146.550522 \n",
       "L 168.5125 146.550522 \n",
       "L 165.460938 146.550522 \n",
       "L 162.409375 146.550522 \n",
       "L 159.357813 146.550521 \n",
       "L 156.30625 146.550521 \n",
       "L 153.254688 146.550521 \n",
       "L 150.203125 146.55052 \n",
       "L 147.151563 146.55052 \n",
       "L 144.1 146.550519 \n",
       "L 141.048438 146.550519 \n",
       "L 137.996875 146.550519 \n",
       "L 134.945312 146.550518 \n",
       "L 131.89375 146.550518 \n",
       "L 128.842188 146.550517 \n",
       "L 125.790625 146.550517 \n",
       "L 122.739063 146.550516 \n",
       "L 119.6875 146.550516 \n",
       "L 116.635938 146.550515 \n",
       "L 113.584375 146.550515 \n",
       "L 110.532812 146.550514 \n",
       "L 107.48125 146.550514 \n",
       "L 104.429688 146.550513 \n",
       "L 101.378125 146.550512 \n",
       "L 98.326563 146.550512 \n",
       "L 95.275 146.550511 \n",
       "L 92.223438 146.550511 \n",
       "L 89.171875 146.55051 \n",
       "L 86.120313 146.550509 \n",
       "L 83.06875 146.550509 \n",
       "L 80.017188 146.550508 \n",
       "L 76.965625 146.550507 \n",
       "L 73.914062 146.550507 \n",
       "L 70.8625 146.550506 \n",
       "L 67.810937 146.550505 \n",
       "L 64.759375 146.550504 \n",
       "L 61.707813 146.550504 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_10\">\n",
       "    <path clip-path=\"url(#pb767c309e6)\" d=\"M 76.965625 67.426859 \n",
       "L 95.275 67.199617 \n",
       "L 111.38725 67.517756 \n",
       "L 125.56603 67.072362 \n",
       "L 138.043356 67.695913 \n",
       "L 149.023404 66.822942 \n",
       "L 158.685845 68.045102 \n",
       "L 167.188794 66.334077 \n",
       "L 174.671389 68.729512 \n",
       "L 181.256072 65.375904 \n",
       "L 187.050593 70.070955 \n",
       "L 192.149772 63.497883 \n",
       "L 196.637049 72.700184 \n",
       "L 200.585853 59.816963 \n",
       "L 204.060801 77.853472 \n",
       "L 207.118755 52.60236 \n",
       "L 209.809754 87.953917 \n",
       "L 212.177834 38.461737 \n",
       "L 214.261744 107.750789 \n",
       "L 216.095585 10.746116 \n",
       "L 217.709364 146.552658 \n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    <defs>\n",
       "     <path d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" id=\"m2e2b86e16e\" style=\"stroke:#ff7f0e;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#pb767c309e6)\">\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"76.965625\" xlink:href=\"#m2e2b86e16e\" y=\"67.426859\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"95.275\" xlink:href=\"#m2e2b86e16e\" y=\"67.199617\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"111.38725\" xlink:href=\"#m2e2b86e16e\" y=\"67.517756\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"125.56603\" xlink:href=\"#m2e2b86e16e\" y=\"67.072362\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"138.043356\" xlink:href=\"#m2e2b86e16e\" y=\"67.695913\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"149.023404\" xlink:href=\"#m2e2b86e16e\" y=\"66.822942\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"158.685845\" xlink:href=\"#m2e2b86e16e\" y=\"68.045102\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"167.188794\" xlink:href=\"#m2e2b86e16e\" y=\"66.334077\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"174.671389\" xlink:href=\"#m2e2b86e16e\" y=\"68.729512\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"181.256072\" xlink:href=\"#m2e2b86e16e\" y=\"65.375904\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"187.050593\" xlink:href=\"#m2e2b86e16e\" y=\"70.070955\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"192.149772\" xlink:href=\"#m2e2b86e16e\" y=\"63.497883\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"196.637049\" xlink:href=\"#m2e2b86e16e\" y=\"72.700184\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"200.585853\" xlink:href=\"#m2e2b86e16e\" y=\"59.816963\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"204.060801\" xlink:href=\"#m2e2b86e16e\" y=\"77.853472\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"207.118755\" xlink:href=\"#m2e2b86e16e\" y=\"52.60236\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"209.809754\" xlink:href=\"#m2e2b86e16e\" y=\"87.953917\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"212.177834\" xlink:href=\"#m2e2b86e16e\" y=\"38.461737\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"214.261744\" xlink:href=\"#m2e2b86e16e\" y=\"107.750789\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"216.095585\" xlink:href=\"#m2e2b86e16e\" y=\"10.746116\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"217.709364\" xlink:href=\"#m2e2b86e16e\" y=\"146.552658\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 61.707813 146.6 \n",
       "L 61.707813 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 257.007812 146.6 \n",
       "L 257.007812 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 61.707813 146.6 \n",
       "L 257.007812 146.6 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 61.707813 10.7 \n",
       "L 257.007812 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pb767c309e6\">\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"61.707813\" y=\"10.7\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "eta = 0.6\n",
    "d2lt.show_trace_2d(f_2d, d2lt.train_2d(gd_2d))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 动量法\n",
    "\n",
    "动量法的提出是为了解决梯度下降的上述问题。由于小批量随机梯度下降比梯度下降更为广义，本章后续讨论将沿用[“小批量随机梯度下降”](minibatch-sgd.ipynb)一节中时间步$t$的小批量随机梯度$\\boldsymbol{g}_t$的定义。设时间步$t$的自变量为$\\boldsymbol{x}_t$，学习率为$\\eta_t$。\n",
    "在时间步$0$，动量法创建速度变量$\\boldsymbol{v}_0$，并将其元素初始化成0。在时间步$t>0$，动量法对每次迭代的步骤做如下修改：\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\boldsymbol{v}_t &\\leftarrow \\gamma \\boldsymbol{v}_{t-1} + \\eta_t \\boldsymbol{g}_t, \\\\\n",
    "\\boldsymbol{x}_t &\\leftarrow \\boldsymbol{x}_{t-1} - \\boldsymbol{v}_t,\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "其中，动量超参数$\\gamma$满足$0 \\leq \\gamma < 1$。当$\\gamma=0$时，动量法等价于小批量随机梯度下降。\n",
    "\n",
    "在解释动量法的数学原理前，让我们先从实验中观察梯度下降在使用动量法后的迭代轨迹。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "5"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 20, x1 -0.062843, x2 0.001202\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 248.620313 184.15625\" width=\"248.620313pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 184.15625 \n",
       "L 248.620313 184.15625 \n",
       "L 248.620313 -0 \n",
       "L 0 -0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 42.620312 146.6 \n",
       "L 237.920313 146.6 \n",
       "L 237.920313 10.7 \n",
       "L 42.620312 10.7 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"m70b026714f\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"88.39375\" xlink:href=\"#m70b026714f\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −4 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-2212\"/>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"DejaVuSans-34\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(81.022656 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"149.425\" xlink:href=\"#m70b026714f\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −2 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-32\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(142.053906 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"210.45625\" xlink:href=\"#m70b026714f\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-30\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(207.275 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_4\">\n",
       "     <!-- x1 -->\n",
       "     <defs>\n",
       "      <path d=\"M 54.890625 54.6875 \n",
       "L 35.109375 28.078125 \n",
       "L 55.90625 0 \n",
       "L 45.3125 0 \n",
       "L 29.390625 21.484375 \n",
       "L 13.484375 0 \n",
       "L 2.875 0 \n",
       "L 24.125 28.609375 \n",
       "L 4.6875 54.6875 \n",
       "L 15.28125 54.6875 \n",
       "L 29.78125 35.203125 \n",
       "L 44.28125 54.6875 \n",
       "z\n",
       "\" id=\"DejaVuSans-78\"/>\n",
       "      <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-31\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(134.129687 174.876562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "      <use x=\"59.179688\" xlink:href=\"#DejaVuSans-31\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"m5f3c7c506c\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620312\" xlink:href=\"#m5f3c7c506c\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- −3 -->\n",
       "      <defs>\n",
       "       <path d=\"M 40.578125 39.3125 \n",
       "Q 47.65625 37.796875 51.625 33 \n",
       "Q 55.609375 28.21875 55.609375 21.1875 \n",
       "Q 55.609375 10.40625 48.1875 4.484375 \n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \n",
       "Q 12.796875 0.390625 7.625 2.203125 \n",
       "L 7.625 11.71875 \n",
       "Q 11.71875 9.328125 16.59375 8.109375 \n",
       "Q 21.484375 6.890625 26.8125 6.890625 \n",
       "Q 36.078125 6.890625 40.9375 10.546875 \n",
       "Q 45.796875 14.203125 45.796875 21.1875 \n",
       "Q 45.796875 27.640625 41.28125 31.265625 \n",
       "Q 36.765625 34.90625 28.71875 34.90625 \n",
       "L 20.21875 34.90625 \n",
       "L 20.21875 43.015625 \n",
       "L 29.109375 43.015625 \n",
       "Q 36.375 43.015625 40.234375 45.921875 \n",
       "Q 44.09375 48.828125 44.09375 54.296875 \n",
       "Q 44.09375 59.90625 40.109375 62.90625 \n",
       "Q 36.140625 65.921875 28.71875 65.921875 \n",
       "Q 24.65625 65.921875 20.015625 65.03125 \n",
       "Q 15.375 64.15625 9.8125 62.3125 \n",
       "L 9.8125 71.09375 \n",
       "Q 15.4375 72.65625 20.34375 73.4375 \n",
       "Q 25.25 74.21875 29.59375 74.21875 \n",
       "Q 40.828125 74.21875 47.359375 69.109375 \n",
       "Q 53.90625 64.015625 53.90625 55.328125 \n",
       "Q 53.90625 49.265625 50.4375 45.09375 \n",
       "Q 46.96875 40.921875 40.578125 39.3125 \n",
       "z\n",
       "\" id=\"DejaVuSans-33\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(20.878125 150.399219)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620312\" xlink:href=\"#m5f3c7c506c\" y=\"119.952941\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- −2 -->\n",
       "      <g transform=\"translate(20.878125 123.75216)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620312\" xlink:href=\"#m5f3c7c506c\" y=\"93.305882\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- −1 -->\n",
       "      <g transform=\"translate(20.878125 97.105101)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-31\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620312\" xlink:href=\"#m5f3c7c506c\" y=\"66.658824\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(29.257812 70.458042)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620312\" xlink:href=\"#m5f3c7c506c\" y=\"40.011765\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 1 -->\n",
       "      <g transform=\"translate(29.257812 43.810983)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620312\" xlink:href=\"#m5f3c7c506c\" y=\"13.364706\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(29.257812 17.163925)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- x2 -->\n",
       "     <g transform=\"translate(14.798437 84.790625)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "      <use x=\"59.179688\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_1\">\n",
       "    <path clip-path=\"url(#p7123913bac)\" d=\"M 237.920313 98.843141 \n",
       "L 222.6625 99.189553 \n",
       "L 207.404688 99.269494 \n",
       "L 192.146875 99.082965 \n",
       "L 176.889063 98.629501 \n",
       "L 164.682813 98.027046 \n",
       "L 152.476563 97.23922 \n",
       "L 140.270313 96.266023 \n",
       "L 131.115625 95.361513 \n",
       "L 118.909375 93.940336 \n",
       "L 109.754688 92.68178 \n",
       "L 97.100875 90.641176 \n",
       "L 85.342188 88.282128 \n",
       "L 82.290625 87.585647 \n",
       "L 73.135938 85.26052 \n",
       "L 63.98125 82.356364 \n",
       "L 57.878125 79.982353 \n",
       "L 52.486044 77.317647 \n",
       "L 48.409725 74.652941 \n",
       "L 45.559891 71.988235 \n",
       "L 43.880132 69.323529 \n",
       "L 43.320212 66.658824 \n",
       "L 43.880132 63.994118 \n",
       "L 45.671875 61.222824 \n",
       "L 48.723438 58.455336 \n",
       "L 52.486044 56 \n",
       "L 57.878125 53.335294 \n",
       "L 64.752171 50.670588 \n",
       "L 73.307374 48.005882 \n",
       "L 83.908321 45.341176 \n",
       "L 94.496875 43.178062 \n",
       "L 100.6 42.08743 \n",
       "L 113.96875 40.011765 \n",
       "L 125.0125 38.641345 \n",
       "L 137.21875 37.323887 \n",
       "L 149.425 36.304348 \n",
       "L 161.63125 35.470179 \n",
       "L 173.8375 34.821381 \n",
       "L 186.04375 34.383906 \n",
       "L 201.301563 34.090788 \n",
       "L 216.559375 34.064141 \n",
       "L 231.817188 34.303965 \n",
       "L 237.920313 34.474506 \n",
       "L 237.920313 34.474506 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_2\">\n",
       "    <path clip-path=\"url(#p7123913bac)\" d=\"M 237.920313 112.487958 \n",
       "L 234.86875 112.552672 \n",
       "L 231.817188 112.609773 \n",
       "L 228.765625 112.659261 \n",
       "L 225.714063 112.701134 \n",
       "L 222.6625 112.735395 \n",
       "L 219.610938 112.762042 \n",
       "L 216.559375 112.781076 \n",
       "L 213.507813 112.792496 \n",
       "L 210.45625 112.796303 \n",
       "L 207.404688 112.792496 \n",
       "L 204.353125 112.781076 \n",
       "L 201.301563 112.762042 \n",
       "L 198.25 112.735395 \n",
       "L 195.198438 112.701134 \n",
       "L 192.146875 112.659261 \n",
       "L 189.095313 112.609773 \n",
       "L 186.04375 112.552672 \n",
       "L 182.992188 112.487958 \n",
       "L 179.940625 112.41563 \n",
       "L 176.889063 112.335689 \n",
       "L 173.8375 112.248134 \n",
       "L 170.785938 112.152966 \n",
       "L 167.734375 112.050185 \n",
       "L 165.208944 111.958824 \n",
       "L 164.682813 111.938636 \n",
       "L 161.63125 111.813476 \n",
       "L 158.579688 111.680241 \n",
       "L 155.528125 111.53893 \n",
       "L 152.476563 111.389545 \n",
       "L 149.425 111.232086 \n",
       "L 146.373438 111.066551 \n",
       "L 143.321875 110.892941 \n",
       "L 140.270313 110.711257 \n",
       "L 137.21875 110.521497 \n",
       "L 134.167188 110.323663 \n",
       "L 131.115625 110.117754 \n",
       "L 128.064063 109.90377 \n",
       "L 125.0125 109.681711 \n",
       "L 121.960938 109.451578 \n",
       "L 119.943803 109.294118 \n",
       "L 118.909375 109.208159 \n",
       "L 115.857813 108.945987 \n",
       "L 112.80625 108.675218 \n",
       "L 109.754688 108.395854 \n",
       "L 106.703125 108.107894 \n",
       "L 103.651563 107.811338 \n",
       "L 100.6 107.506186 \n",
       "L 97.548438 107.192438 \n",
       "L 94.496875 106.870095 \n",
       "L 92.277557 106.629412 \n",
       "L 91.445313 106.532931 \n",
       "L 88.39375 106.16998 \n",
       "L 85.342188 105.79784 \n",
       "L 82.290625 105.416511 \n",
       "L 79.239063 105.025994 \n",
       "L 76.1875 104.626288 \n",
       "L 73.135938 104.217394 \n",
       "L 71.291587 103.964706 \n",
       "L 70.084375 103.787059 \n",
       "L 67.032813 103.328137 \n",
       "L 63.98125 102.859346 \n",
       "L 60.929688 102.380686 \n",
       "L 57.878125 101.892157 \n",
       "L 54.826563 101.393758 \n",
       "L 54.263653 101.3 \n",
       "L 51.775 100.852329 \n",
       "L 48.723438 100.292741 \n",
       "L 45.671875 99.722494 \n",
       "L 42.620313 99.141588 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#p7123913bac)\" d=\"M 42.620313 34.176059 \n",
       "L 45.671875 33.595153 \n",
       "L 48.723438 33.024906 \n",
       "L 51.775 32.465318 \n",
       "L 54.263653 32.017647 \n",
       "L 54.826563 31.923889 \n",
       "L 57.878125 31.42549 \n",
       "L 60.929688 30.936961 \n",
       "L 63.98125 30.458301 \n",
       "L 67.032813 29.98951 \n",
       "L 70.084375 29.530588 \n",
       "L 71.291587 29.352941 \n",
       "L 73.135938 29.100254 \n",
       "L 76.1875 28.691359 \n",
       "L 79.239063 28.291653 \n",
       "L 82.290625 27.901136 \n",
       "L 85.342188 27.519807 \n",
       "L 88.39375 27.147667 \n",
       "L 91.445313 26.784716 \n",
       "L 92.277557 26.688235 \n",
       "L 94.496875 26.447552 \n",
       "L 97.548438 26.125209 \n",
       "L 100.6 25.811461 \n",
       "L 103.651563 25.506309 \n",
       "L 106.703125 25.209753 \n",
       "L 109.754688 24.921793 \n",
       "L 112.80625 24.642429 \n",
       "L 115.857813 24.37166 \n",
       "L 118.909375 24.109488 \n",
       "L 119.943803 24.023529 \n",
       "L 121.960938 23.86607 \n",
       "L 125.0125 23.635936 \n",
       "L 128.064063 23.413877 \n",
       "L 131.115625 23.199893 \n",
       "L 134.167188 22.993984 \n",
       "L 137.21875 22.79615 \n",
       "L 140.270313 22.60639 \n",
       "L 143.321875 22.424706 \n",
       "L 146.373438 22.251096 \n",
       "L 149.425 22.085561 \n",
       "L 152.476563 21.928102 \n",
       "L 155.528125 21.778717 \n",
       "L 158.579688 21.637406 \n",
       "L 161.63125 21.504171 \n",
       "L 164.682813 21.379011 \n",
       "L 165.208944 21.358824 \n",
       "L 167.734375 21.267462 \n",
       "L 170.785938 21.164681 \n",
       "L 173.8375 21.069513 \n",
       "L 176.889063 20.981958 \n",
       "L 179.940625 20.902017 \n",
       "L 182.992188 20.829689 \n",
       "L 186.04375 20.764975 \n",
       "L 189.095313 20.707874 \n",
       "L 192.146875 20.658387 \n",
       "L 195.198438 20.616513 \n",
       "L 198.25 20.582252 \n",
       "L 201.301563 20.555605 \n",
       "L 204.353125 20.536571 \n",
       "L 207.404688 20.525151 \n",
       "L 210.45625 20.521345 \n",
       "L 213.507813 20.525151 \n",
       "L 216.559375 20.536571 \n",
       "L 219.610938 20.555605 \n",
       "L 222.6625 20.582252 \n",
       "L 225.714063 20.616513 \n",
       "L 228.765625 20.658387 \n",
       "L 231.817188 20.707874 \n",
       "L 234.86875 20.764975 \n",
       "L 237.920313 20.829689 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_3\">\n",
       "    <path clip-path=\"url(#p7123913bac)\" d=\"M 237.920313 122.924398 \n",
       "L 234.86875 122.977073 \n",
       "L 231.817188 123.02355 \n",
       "L 228.765625 123.06383 \n",
       "L 225.714063 123.097914 \n",
       "L 222.6625 123.1258 \n",
       "L 219.610938 123.14749 \n",
       "L 216.559375 123.162982 \n",
       "L 213.507813 123.172278 \n",
       "L 210.45625 123.175376 \n",
       "L 207.404688 123.172278 \n",
       "L 204.353125 123.162982 \n",
       "L 201.301563 123.14749 \n",
       "L 198.25 123.1258 \n",
       "L 195.198438 123.097914 \n",
       "L 192.146875 123.06383 \n",
       "L 189.095313 123.02355 \n",
       "L 186.04375 122.977073 \n",
       "L 182.992188 122.924398 \n",
       "L 179.940625 122.865527 \n",
       "L 176.889063 122.800458 \n",
       "L 173.8375 122.729193 \n",
       "L 170.785938 122.651731 \n",
       "L 169.542708 122.617647 \n",
       "L 167.734375 122.565653 \n",
       "L 164.682813 122.471413 \n",
       "L 161.63125 122.370674 \n",
       "L 158.579688 122.263436 \n",
       "L 155.528125 122.149699 \n",
       "L 152.476563 122.029462 \n",
       "L 149.425 121.902726 \n",
       "L 146.373438 121.769491 \n",
       "L 143.321875 121.629756 \n",
       "L 140.270313 121.483522 \n",
       "L 137.21875 121.330789 \n",
       "L 134.167188 121.171557 \n",
       "L 131.115625 121.005825 \n",
       "L 128.064063 120.833594 \n",
       "L 125.0125 120.654864 \n",
       "L 121.960938 120.469634 \n",
       "L 118.909375 120.277905 \n",
       "L 115.857813 120.079677 \n",
       "L 113.96875 119.952941 \n",
       "L 112.80625 119.87095 \n",
       "L 109.754688 119.648891 \n",
       "L 106.703125 119.42 \n",
       "L 103.651563 119.184276 \n",
       "L 100.6 118.941719 \n",
       "L 97.548438 118.69233 \n",
       "L 94.496875 118.436109 \n",
       "L 91.445313 118.173054 \n",
       "L 88.39375 117.903167 \n",
       "L 85.342188 117.626448 \n",
       "L 82.290625 117.342896 \n",
       "L 81.716213 117.288235 \n",
       "L 79.239063 117.039769 \n",
       "L 76.1875 116.726486 \n",
       "L 73.135938 116.406002 \n",
       "L 70.084375 116.078315 \n",
       "L 67.032813 115.743426 \n",
       "L 63.98125 115.401335 \n",
       "L 60.929688 115.052043 \n",
       "L 57.878125 114.695548 \n",
       "L 57.273855 114.623529 \n",
       "L 54.826563 114.315185 \n",
       "L 51.775 113.923092 \n",
       "L 48.723438 113.523387 \n",
       "L 45.671875 113.116067 \n",
       "L 42.620313 112.701134 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#p7123913bac)\" d=\"M 42.620313 20.616513 \n",
       "L 45.671875 20.20158 \n",
       "L 48.723438 19.794261 \n",
       "L 51.775 19.394555 \n",
       "L 54.826563 19.002462 \n",
       "L 57.273855 18.694118 \n",
       "L 57.878125 18.622099 \n",
       "L 60.929688 18.265604 \n",
       "L 63.98125 17.916312 \n",
       "L 67.032813 17.574221 \n",
       "L 70.084375 17.239332 \n",
       "L 73.135938 16.911645 \n",
       "L 76.1875 16.591161 \n",
       "L 79.239063 16.277878 \n",
       "L 81.716213 16.029412 \n",
       "L 82.290625 15.974751 \n",
       "L 85.342188 15.691199 \n",
       "L 88.39375 15.41448 \n",
       "L 91.445313 15.144593 \n",
       "L 94.496875 14.881538 \n",
       "L 97.548438 14.625317 \n",
       "L 100.6 14.375928 \n",
       "L 103.651563 14.133371 \n",
       "L 106.703125 13.897647 \n",
       "L 109.754688 13.668756 \n",
       "L 112.80625 13.446697 \n",
       "L 113.96875 13.364706 \n",
       "L 115.857813 13.23797 \n",
       "L 118.909375 13.039742 \n",
       "L 121.960938 12.848013 \n",
       "L 125.0125 12.662783 \n",
       "L 128.064063 12.484053 \n",
       "L 131.115625 12.311822 \n",
       "L 134.167188 12.14609 \n",
       "L 137.21875 11.986858 \n",
       "L 140.270313 11.834125 \n",
       "L 143.321875 11.687891 \n",
       "L 146.373438 11.548156 \n",
       "L 149.425 11.414921 \n",
       "L 152.476563 11.288185 \n",
       "L 155.528125 11.167948 \n",
       "L 158.579688 11.054211 \n",
       "L 161.63125 10.946973 \n",
       "L 164.682813 10.846234 \n",
       "L 167.734375 10.751994 \n",
       "L 169.542708 10.7 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_4\">\n",
       "    <path clip-path=\"url(#p7123913bac)\" d=\"M 237.920313 131.696681 \n",
       "L 234.86875 131.742905 \n",
       "L 231.817188 131.783691 \n",
       "L 228.765625 131.81904 \n",
       "L 225.714063 131.84895 \n",
       "L 222.6625 131.873421 \n",
       "L 219.610938 131.892455 \n",
       "L 216.559375 131.90605 \n",
       "L 213.507813 131.914208 \n",
       "L 210.45625 131.916927 \n",
       "L 207.404688 131.914208 \n",
       "L 204.353125 131.90605 \n",
       "L 201.301563 131.892455 \n",
       "L 198.25 131.873421 \n",
       "L 195.198438 131.84895 \n",
       "L 192.146875 131.81904 \n",
       "L 189.095313 131.783691 \n",
       "L 186.04375 131.742905 \n",
       "L 182.992188 131.696681 \n",
       "L 179.940625 131.645018 \n",
       "L 176.889063 131.587917 \n",
       "L 173.8375 131.525378 \n",
       "L 170.785938 131.457401 \n",
       "L 167.734375 131.383986 \n",
       "L 164.682813 131.305132 \n",
       "L 161.63125 131.22084 \n",
       "L 158.579688 131.13111 \n",
       "L 155.528125 131.035942 \n",
       "L 152.476563 130.935336 \n",
       "L 149.425 130.829292 \n",
       "L 146.373438 130.717809 \n",
       "L 143.605741 130.611765 \n",
       "L 143.321875 130.600426 \n",
       "L 140.270313 130.47286 \n",
       "L 137.21875 130.339625 \n",
       "L 134.167188 130.20072 \n",
       "L 131.115625 130.056145 \n",
       "L 128.064063 129.905901 \n",
       "L 125.0125 129.749987 \n",
       "L 121.960938 129.588404 \n",
       "L 118.909375 129.421151 \n",
       "L 115.857813 129.248229 \n",
       "L 112.80625 129.069637 \n",
       "L 109.754688 128.885375 \n",
       "L 106.703125 128.695444 \n",
       "L 103.651563 128.499844 \n",
       "L 100.6 128.298573 \n",
       "L 97.548438 128.091633 \n",
       "L 95.473375 127.947059 \n",
       "L 94.496875 127.876 \n",
       "L 91.445313 127.64802 \n",
       "L 88.39375 127.414118 \n",
       "L 85.342188 127.174294 \n",
       "L 82.290625 126.928549 \n",
       "L 79.239063 126.676882 \n",
       "L 76.1875 126.419294 \n",
       "L 73.135938 126.155784 \n",
       "L 70.084375 125.886353 \n",
       "L 67.032813 125.611 \n",
       "L 63.98125 125.329725 \n",
       "L 63.477899 125.282353 \n",
       "L 60.929688 125.031375 \n",
       "L 57.878125 124.724624 \n",
       "L 54.826563 124.411676 \n",
       "L 51.775 124.092531 \n",
       "L 48.723438 123.767189 \n",
       "L 45.671875 123.43565 \n",
       "L 42.620313 123.097914 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_5\">\n",
       "    <path clip-path=\"url(#p7123913bac)\" d=\"M 237.920313 139.427096 \n",
       "L 234.86875 139.468278 \n",
       "L 231.817188 139.504615 \n",
       "L 228.765625 139.536107 \n",
       "L 225.714063 139.562754 \n",
       "L 222.6625 139.584556 \n",
       "L 219.610938 139.601513 \n",
       "L 216.559375 139.613626 \n",
       "L 213.507813 139.620893 \n",
       "L 210.45625 139.623316 \n",
       "L 207.404688 139.620893 \n",
       "L 204.353125 139.613626 \n",
       "L 201.301563 139.601513 \n",
       "L 198.25 139.584556 \n",
       "L 195.198438 139.562754 \n",
       "L 192.146875 139.536107 \n",
       "L 189.095313 139.504615 \n",
       "L 186.04375 139.468278 \n",
       "L 182.992188 139.427096 \n",
       "L 179.940625 139.38107 \n",
       "L 176.889063 139.330198 \n",
       "L 173.8375 139.274481 \n",
       "L 170.785938 139.21392 \n",
       "L 167.734375 139.148513 \n",
       "L 164.682813 139.078262 \n",
       "L 161.63125 139.003166 \n",
       "L 158.579688 138.923225 \n",
       "L 155.528125 138.838439 \n",
       "L 152.476563 138.748807 \n",
       "L 149.425 138.654332 \n",
       "L 147.936433 138.605882 \n",
       "L 146.373438 138.553091 \n",
       "L 143.321875 138.444994 \n",
       "L 140.270313 138.33187 \n",
       "L 137.21875 138.213718 \n",
       "L 134.167188 138.090538 \n",
       "L 131.115625 137.962331 \n",
       "L 128.064063 137.829095 \n",
       "L 125.0125 137.690832 \n",
       "L 121.960938 137.547542 \n",
       "L 118.909375 137.399223 \n",
       "L 115.857813 137.245877 \n",
       "L 112.80625 137.087503 \n",
       "L 109.754688 136.924101 \n",
       "L 106.703125 136.755671 \n",
       "L 103.651563 136.582214 \n",
       "L 100.6 136.403729 \n",
       "L 97.548438 136.220216 \n",
       "L 94.496875 136.031676 \n",
       "L 93.07017 135.941176 \n",
       "L 91.445313 135.834066 \n",
       "L 88.39375 135.627682 \n",
       "L 85.342188 135.416073 \n",
       "L 82.290625 135.199239 \n",
       "L 79.239063 134.97718 \n",
       "L 76.1875 134.749896 \n",
       "L 73.135938 134.517388 \n",
       "L 70.084375 134.279654 \n",
       "L 67.032813 134.036696 \n",
       "L 63.98125 133.788512 \n",
       "L 60.929688 133.535104 \n",
       "L 57.878125 133.276471 \n",
       "L 57.878125 133.276471 \n",
       "L 54.826563 133.001843 \n",
       "L 51.775 132.721777 \n",
       "L 48.723438 132.436273 \n",
       "L 45.671875 132.14533 \n",
       "L 42.620313 131.84895 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_6\">\n",
       "    <path clip-path=\"url(#p7123913bac)\" d=\"M 210.45625 146.6 \n",
       "L 207.404688 146.597742 \n",
       "L 204.353125 146.590967 \n",
       "L 201.301563 146.579676 \n",
       "L 198.25 146.563868 \n",
       "L 195.198438 146.543544 \n",
       "L 192.146875 146.518704 \n",
       "L 189.095313 146.489347 \n",
       "L 186.04375 146.455474 \n",
       "L 182.992188 146.417084 \n",
       "L 179.940625 146.374177 \n",
       "L 176.889063 146.326755 \n",
       "L 173.8375 146.274816 \n",
       "L 170.785938 146.21836 \n",
       "L 167.734375 146.157388 \n",
       "L 164.682813 146.091899 \n",
       "L 161.63125 146.021894 \n",
       "L 158.579688 145.947373 \n",
       "L 155.528125 145.868335 \n",
       "L 152.476563 145.784781 \n",
       "L 149.425 145.69671 \n",
       "L 146.373438 145.604123 \n",
       "L 143.321875 145.507019 \n",
       "L 140.270313 145.405399 \n",
       "L 137.21875 145.299262 \n",
       "L 134.167188 145.188609 \n",
       "L 131.115625 145.07344 \n",
       "L 128.064063 144.953754 \n",
       "L 125.0125 144.829551 \n",
       "L 121.960938 144.700833 \n",
       "L 118.909375 144.567597 \n",
       "L 115.857813 144.429845 \n",
       "L 112.80625 144.287577 \n",
       "L 109.754688 144.140793 \n",
       "L 106.703125 143.989492 \n",
       "L 105.641712 143.935294 \n",
       "L 103.651563 143.830108 \n",
       "L 100.6 143.664149 \n",
       "L 97.548438 143.493514 \n",
       "L 94.496875 143.318204 \n",
       "L 91.445313 143.13822 \n",
       "L 88.39375 142.95356 \n",
       "L 85.342188 142.764226 \n",
       "L 82.290625 142.570217 \n",
       "L 79.239063 142.371533 \n",
       "L 76.1875 142.168173 \n",
       "L 73.135938 141.960139 \n",
       "L 70.084375 141.74743 \n",
       "L 67.032813 141.530046 \n",
       "L 63.98125 141.307988 \n",
       "L 63.477899 141.270588 \n",
       "L 60.929688 141.074369 \n",
       "L 57.878125 140.834545 \n",
       "L 54.826563 140.589877 \n",
       "L 51.775 140.340364 \n",
       "L 48.723438 140.086005 \n",
       "L 45.671875 139.826802 \n",
       "L 42.620313 139.562754 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#p7123913bac)\" d=\"M 237.920313 146.417084 \n",
       "L 234.86875 146.455474 \n",
       "L 231.817188 146.489347 \n",
       "L 228.765625 146.518704 \n",
       "L 225.714063 146.543544 \n",
       "L 222.6625 146.563868 \n",
       "L 219.610938 146.579676 \n",
       "L 216.559375 146.590967 \n",
       "L 213.507813 146.597742 \n",
       "L 210.45625 146.6 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_7\">\n",
       "    <path clip-path=\"url(#p7123913bac)\" d=\"M 43.320212 146.6 \n",
       "L 42.620313 146.543544 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_10\">\n",
       "    <path clip-path=\"url(#p7123913bac)\" d=\"M 57.878125 119.952941 \n",
       "L 70.084375 34.682353 \n",
       "L 87.41725 43.209412 \n",
       "L 105.926808 84.992 \n",
       "L 123.543942 76.550212 \n",
       "L 139.305493 56.503096 \n",
       "L 152.87833 62.728702 \n",
       "L 164.270982 72.129699 \n",
       "L 173.662129 68.076797 \n",
       "L 181.301232 63.781588 \n",
       "L 187.453185 66.23756 \n",
       "L 192.369407 68.139567 \n",
       "L 196.274465 66.721381 \n",
       "L 199.361537 65.912196 \n",
       "L 201.79265 66.702208 \n",
       "L 203.701295 67.027799 \n",
       "L 205.196013 66.600234 \n",
       "L 206.364192 66.480195 \n",
       "L 207.275645 66.705981 \n",
       "L 207.985821 66.743422 \n",
       "L 208.538543 66.626785 \n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    <defs>\n",
       "     <path d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" id=\"mab3192e9e4\" style=\"stroke:#ff7f0e;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#p7123913bac)\">\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"57.878125\" xlink:href=\"#mab3192e9e4\" y=\"119.952941\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"70.084375\" xlink:href=\"#mab3192e9e4\" y=\"34.682353\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"87.41725\" xlink:href=\"#mab3192e9e4\" y=\"43.209412\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"105.926808\" xlink:href=\"#mab3192e9e4\" y=\"84.992\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"123.543942\" xlink:href=\"#mab3192e9e4\" y=\"76.550212\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"139.305493\" xlink:href=\"#mab3192e9e4\" y=\"56.503096\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"152.87833\" xlink:href=\"#mab3192e9e4\" y=\"62.728702\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"164.270982\" xlink:href=\"#mab3192e9e4\" y=\"72.129699\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"173.662129\" xlink:href=\"#mab3192e9e4\" y=\"68.076797\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"181.301232\" xlink:href=\"#mab3192e9e4\" y=\"63.781588\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"187.453185\" xlink:href=\"#mab3192e9e4\" y=\"66.23756\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"192.369407\" xlink:href=\"#mab3192e9e4\" y=\"68.139567\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"196.274465\" xlink:href=\"#mab3192e9e4\" y=\"66.721381\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"199.361537\" xlink:href=\"#mab3192e9e4\" y=\"65.912196\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"201.79265\" xlink:href=\"#mab3192e9e4\" y=\"66.702208\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"203.701295\" xlink:href=\"#mab3192e9e4\" y=\"67.027799\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"205.196013\" xlink:href=\"#mab3192e9e4\" y=\"66.600234\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"206.364192\" xlink:href=\"#mab3192e9e4\" y=\"66.480195\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"207.275645\" xlink:href=\"#mab3192e9e4\" y=\"66.705981\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"207.985821\" xlink:href=\"#mab3192e9e4\" y=\"66.743422\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"208.538543\" xlink:href=\"#mab3192e9e4\" y=\"66.626785\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 42.620312 146.6 \n",
       "L 42.620312 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 237.920313 146.6 \n",
       "L 237.920313 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 42.620313 146.6 \n",
       "L 237.920313 146.6 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 42.620313 10.7 \n",
       "L 237.920313 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p7123913bac\">\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"42.620312\" y=\"10.7\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def momentum_2d(x1, x2, v1, v2):\n",
    "    v1 = gamma * v1 + eta * 0.2 * x1\n",
    "    v2 = gamma * v2 + eta * 4 * x2\n",
    "    return x1 - v1, x2 - v2, v1, v2\n",
    "\n",
    "eta, gamma = 0.4, 0.5\n",
    "d2lt.show_trace_2d(f_2d, d2lt.train_2d(momentum_2d))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到使用较小的学习率$\\eta=0.4$和动量超参数$\\gamma=0.5$时，动量法在竖直方向上的移动更加平滑，且在水平方向上更快逼近最优解。下面使用较大的学习率$\\eta=0.6$，此时自变量也不再发散。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "11"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 20, x1 0.007188, x2 0.002553\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 248.620313 184.15625\" width=\"248.620313pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 184.15625 \n",
       "L 248.620313 184.15625 \n",
       "L 248.620313 -0 \n",
       "L 0 -0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 42.620312 146.6 \n",
       "L 237.920313 146.6 \n",
       "L 237.920313 10.7 \n",
       "L 42.620312 10.7 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"mc609f78e1d\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"88.39375\" xlink:href=\"#mc609f78e1d\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −4 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-2212\"/>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"DejaVuSans-34\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(81.022656 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"149.425\" xlink:href=\"#mc609f78e1d\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −2 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-32\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(142.053906 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"210.45625\" xlink:href=\"#mc609f78e1d\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-30\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(207.275 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_4\">\n",
       "     <!-- x1 -->\n",
       "     <defs>\n",
       "      <path d=\"M 54.890625 54.6875 \n",
       "L 35.109375 28.078125 \n",
       "L 55.90625 0 \n",
       "L 45.3125 0 \n",
       "L 29.390625 21.484375 \n",
       "L 13.484375 0 \n",
       "L 2.875 0 \n",
       "L 24.125 28.609375 \n",
       "L 4.6875 54.6875 \n",
       "L 15.28125 54.6875 \n",
       "L 29.78125 35.203125 \n",
       "L 44.28125 54.6875 \n",
       "z\n",
       "\" id=\"DejaVuSans-78\"/>\n",
       "      <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-31\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(134.129687 174.876562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "      <use x=\"59.179688\" xlink:href=\"#DejaVuSans-31\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"m7aa3e23bbd\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620312\" xlink:href=\"#m7aa3e23bbd\" y=\"126.316418\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- −2 -->\n",
       "      <g transform=\"translate(20.878125 130.115637)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620312\" xlink:href=\"#m7aa3e23bbd\" y=\"85.749254\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(29.257812 89.548472)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"42.620312\" xlink:href=\"#m7aa3e23bbd\" y=\"45.18209\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(29.257812 48.981308)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_8\">\n",
       "     <!-- x2 -->\n",
       "     <g transform=\"translate(14.798437 84.790625)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-78\"/>\n",
       "      <use x=\"59.179688\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_1\">\n",
       "    <path clip-path=\"url(#pd359cbc70d)\" d=\"M 237.920313 114.142512 \n",
       "L 237.740809 114.146269 \n",
       "L 234.86875 114.202223 \n",
       "L 231.817188 114.254681 \n",
       "L 228.765625 114.300144 \n",
       "L 225.714063 114.338613 \n",
       "L 222.6625 114.370087 \n",
       "L 219.610938 114.394568 \n",
       "L 216.559375 114.412054 \n",
       "L 213.507813 114.422545 \n",
       "L 210.45625 114.426042 \n",
       "L 207.404688 114.422545 \n",
       "L 204.353125 114.412054 \n",
       "L 201.301563 114.394568 \n",
       "L 198.25 114.370087 \n",
       "L 195.198438 114.338613 \n",
       "L 192.146875 114.300144 \n",
       "L 189.095313 114.254681 \n",
       "L 186.04375 114.202223 \n",
       "L 183.171691 114.146269 \n",
       "L 182.992188 114.142512 \n",
       "L 179.940625 114.071144 \n",
       "L 176.889063 113.992264 \n",
       "L 173.8375 113.905871 \n",
       "L 170.785938 113.811965 \n",
       "L 167.734375 113.710547 \n",
       "L 164.682813 113.601617 \n",
       "L 161.63125 113.485174 \n",
       "L 158.579688 113.361219 \n",
       "L 155.528125 113.229751 \n",
       "L 152.476563 113.090771 \n",
       "L 149.425 112.944279 \n",
       "L 146.373438 112.790274 \n",
       "L 143.321875 112.628756 \n",
       "L 140.270313 112.459726 \n",
       "L 137.21875 112.283184 \n",
       "L 134.478571 112.11791 \n",
       "L 134.167188 112.097627 \n",
       "L 131.115625 111.890734 \n",
       "L 128.064063 111.675728 \n",
       "L 125.0125 111.452609 \n",
       "L 121.960938 111.221376 \n",
       "L 118.909375 110.98203 \n",
       "L 115.857813 110.73457 \n",
       "L 112.80625 110.478997 \n",
       "L 109.754688 110.21531 \n",
       "L 108.342771 110.089552 \n",
       "L 106.703125 109.930811 \n",
       "L 103.651563 109.626557 \n",
       "L 100.6 109.313485 \n",
       "L 97.548438 108.991593 \n",
       "L 94.496875 108.660883 \n",
       "L 91.445313 108.321353 \n",
       "L 89.166297 108.061194 \n",
       "L 88.39375 107.964606 \n",
       "L 85.342188 107.573422 \n",
       "L 82.290625 107.17258 \n",
       "L 79.239063 106.762079 \n",
       "L 76.1875 106.341919 \n",
       "L 73.993118 106.032836 \n",
       "L 73.135938 105.899391 \n",
       "L 70.084375 105.413653 \n",
       "L 67.032813 104.917239 \n",
       "L 63.98125 104.410149 \n",
       "L 61.590335 104.004478 \n",
       "L 60.929688 103.879197 \n",
       "L 57.878125 103.288586 \n",
       "L 54.826563 102.686045 \n",
       "L 51.775 102.071572 \n",
       "L 51.31 101.976119 \n",
       "L 48.723438 101.374373 \n",
       "L 45.671875 100.650925 \n",
       "L 42.760292 99.947761 \n",
       "L 42.620313 99.908754 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pd359cbc70d)\" d=\"M 42.620313 71.589753 \n",
       "L 42.760292 71.550746 \n",
       "L 45.671875 70.847582 \n",
       "L 48.723438 70.124134 \n",
       "L 51.31 69.522388 \n",
       "L 51.775 69.426936 \n",
       "L 54.826563 68.812463 \n",
       "L 57.878125 68.209921 \n",
       "L 60.929688 67.619311 \n",
       "L 61.590335 67.49403 \n",
       "L 63.98125 67.088358 \n",
       "L 67.032813 66.581269 \n",
       "L 70.084375 66.084855 \n",
       "L 73.135938 65.599116 \n",
       "L 73.993118 65.465672 \n",
       "L 76.1875 65.156588 \n",
       "L 79.239063 64.736429 \n",
       "L 82.290625 64.325928 \n",
       "L 85.342188 63.925085 \n",
       "L 88.39375 63.533902 \n",
       "L 89.166297 63.437313 \n",
       "L 91.445313 63.177154 \n",
       "L 94.496875 62.837625 \n",
       "L 97.548438 62.506914 \n",
       "L 100.6 62.185023 \n",
       "L 103.651563 61.87195 \n",
       "L 106.703125 61.567696 \n",
       "L 108.342771 61.408955 \n",
       "L 109.754688 61.283197 \n",
       "L 112.80625 61.01951 \n",
       "L 115.857813 60.763937 \n",
       "L 118.909375 60.516478 \n",
       "L 121.960938 60.277131 \n",
       "L 125.0125 60.045899 \n",
       "L 128.064063 59.822779 \n",
       "L 131.115625 59.607773 \n",
       "L 134.167188 59.400881 \n",
       "L 134.478571 59.380597 \n",
       "L 137.21875 59.215323 \n",
       "L 140.270313 59.038781 \n",
       "L 143.321875 58.869751 \n",
       "L 146.373438 58.708234 \n",
       "L 149.425 58.554229 \n",
       "L 152.476563 58.407736 \n",
       "L 155.528125 58.268756 \n",
       "L 158.579688 58.137289 \n",
       "L 161.63125 58.013333 \n",
       "L 164.682813 57.896891 \n",
       "L 167.734375 57.78796 \n",
       "L 170.785938 57.686542 \n",
       "L 173.8375 57.592637 \n",
       "L 176.889063 57.506244 \n",
       "L 179.940625 57.427363 \n",
       "L 182.992188 57.355995 \n",
       "L 183.171691 57.352239 \n",
       "L 186.04375 57.296284 \n",
       "L 189.095313 57.243827 \n",
       "L 192.146875 57.198363 \n",
       "L 195.198438 57.159894 \n",
       "L 198.25 57.12842 \n",
       "L 201.301563 57.10394 \n",
       "L 204.353125 57.086454 \n",
       "L 207.404688 57.075962 \n",
       "L 210.45625 57.072465 \n",
       "L 213.507813 57.075962 \n",
       "L 216.559375 57.086454 \n",
       "L 219.610938 57.10394 \n",
       "L 222.6625 57.12842 \n",
       "L 225.714063 57.159894 \n",
       "L 228.765625 57.198363 \n",
       "L 231.817188 57.243827 \n",
       "L 234.86875 57.296284 \n",
       "L 237.740809 57.352239 \n",
       "L 237.920313 57.355995 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_2\">\n",
       "    <path clip-path=\"url(#pd359cbc70d)\" d=\"M 237.920313 126.105781 \n",
       "L 234.86875 126.149989 \n",
       "L 231.817188 126.188995 \n",
       "L 228.765625 126.222801 \n",
       "L 225.714063 126.251406 \n",
       "L 222.6625 126.274811 \n",
       "L 219.610938 126.293014 \n",
       "L 216.559375 126.306016 \n",
       "L 213.507813 126.313817 \n",
       "L 210.45625 126.316418 \n",
       "L 210.45625 126.316418 \n",
       "L 210.45625 126.316418 \n",
       "L 207.404688 126.313817 \n",
       "L 204.353125 126.306016 \n",
       "L 201.301563 126.293014 \n",
       "L 198.25 126.274811 \n",
       "L 195.198438 126.251406 \n",
       "L 192.146875 126.222801 \n",
       "L 189.095313 126.188995 \n",
       "L 186.04375 126.149989 \n",
       "L 182.992188 126.105781 \n",
       "L 179.940625 126.056372 \n",
       "L 176.889063 126.001762 \n",
       "L 173.8375 125.941952 \n",
       "L 170.785938 125.87694 \n",
       "L 167.734375 125.806728 \n",
       "L 164.682813 125.731315 \n",
       "L 161.63125 125.6507 \n",
       "L 158.579688 125.564885 \n",
       "L 155.528125 125.473869 \n",
       "L 152.476563 125.377652 \n",
       "L 149.425 125.276234 \n",
       "L 146.373438 125.169615 \n",
       "L 143.321875 125.057796 \n",
       "L 140.270313 124.940775 \n",
       "L 137.21875 124.818553 \n",
       "L 134.167188 124.691131 \n",
       "L 131.115625 124.558507 \n",
       "L 128.064063 124.420683 \n",
       "L 125.234432 124.28806 \n",
       "L 125.0125 124.277096 \n",
       "L 121.960938 124.120857 \n",
       "L 118.909375 123.959137 \n",
       "L 115.857813 123.791934 \n",
       "L 112.80625 123.61925 \n",
       "L 109.754688 123.441083 \n",
       "L 106.703125 123.257434 \n",
       "L 103.651563 123.068304 \n",
       "L 100.6 122.873691 \n",
       "L 97.548438 122.673596 \n",
       "L 94.496875 122.468019 \n",
       "L 91.484943 122.259701 \n",
       "L 91.445313 122.256804 \n",
       "L 88.39375 122.027889 \n",
       "L 85.342188 121.793179 \n",
       "L 82.290625 121.552674 \n",
       "L 79.239063 121.306373 \n",
       "L 76.1875 121.054277 \n",
       "L 73.135938 120.796386 \n",
       "L 70.084375 120.532699 \n",
       "L 67.032813 120.263217 \n",
       "L 66.679474 120.231343 \n",
       "L 63.98125 119.973189 \n",
       "L 60.929688 119.675081 \n",
       "L 57.878125 119.370828 \n",
       "L 54.826563 119.060427 \n",
       "L 51.775 118.743881 \n",
       "L 48.723438 118.421187 \n",
       "L 46.698569 118.202985 \n",
       "L 45.671875 118.085209 \n",
       "L 42.620313 117.728611 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pd359cbc70d)\" d=\"M 42.620313 53.769896 \n",
       "L 45.671875 53.413298 \n",
       "L 46.698569 53.295522 \n",
       "L 48.723438 53.07732 \n",
       "L 51.775 52.754627 \n",
       "L 54.826563 52.43808 \n",
       "L 57.878125 52.12768 \n",
       "L 60.929688 51.823426 \n",
       "L 63.98125 51.525319 \n",
       "L 66.679474 51.267164 \n",
       "L 67.032813 51.23529 \n",
       "L 70.084375 50.965808 \n",
       "L 73.135938 50.702122 \n",
       "L 76.1875 50.44423 \n",
       "L 79.239063 50.192134 \n",
       "L 82.290625 49.945834 \n",
       "L 85.342188 49.705328 \n",
       "L 88.39375 49.470618 \n",
       "L 91.445313 49.241704 \n",
       "L 91.484943 49.238806 \n",
       "L 94.496875 49.030488 \n",
       "L 97.548438 48.824911 \n",
       "L 100.6 48.624816 \n",
       "L 103.651563 48.430204 \n",
       "L 106.703125 48.241073 \n",
       "L 109.754688 48.057424 \n",
       "L 112.80625 47.879258 \n",
       "L 115.857813 47.706573 \n",
       "L 118.909375 47.539371 \n",
       "L 121.960938 47.37765 \n",
       "L 125.0125 47.221412 \n",
       "L 125.234432 47.210448 \n",
       "L 128.064063 47.077824 \n",
       "L 131.115625 46.94 \n",
       "L 134.167188 46.807377 \n",
       "L 137.21875 46.679954 \n",
       "L 140.270313 46.557732 \n",
       "L 143.321875 46.440712 \n",
       "L 146.373438 46.328892 \n",
       "L 149.425 46.222273 \n",
       "L 152.476563 46.120855 \n",
       "L 155.528125 46.024638 \n",
       "L 158.579688 45.933622 \n",
       "L 161.63125 45.847807 \n",
       "L 164.682813 45.767193 \n",
       "L 167.734375 45.69178 \n",
       "L 170.785938 45.621567 \n",
       "L 173.8375 45.556556 \n",
       "L 176.889063 45.496745 \n",
       "L 179.940625 45.442135 \n",
       "L 182.992188 45.392727 \n",
       "L 186.04375 45.348519 \n",
       "L 189.095313 45.309512 \n",
       "L 192.146875 45.275706 \n",
       "L 195.198438 45.247101 \n",
       "L 198.25 45.223697 \n",
       "L 201.301563 45.205494 \n",
       "L 204.353125 45.192491 \n",
       "L 207.404688 45.18469 \n",
       "L 210.45625 45.18209 \n",
       "L 213.507813 45.18469 \n",
       "L 216.559375 45.192491 \n",
       "L 219.610938 45.205494 \n",
       "L 222.6625 45.223697 \n",
       "L 225.714063 45.247101 \n",
       "L 228.765625 45.275706 \n",
       "L 231.817188 45.309512 \n",
       "L 234.86875 45.348519 \n",
       "L 237.920313 45.392727 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_3\">\n",
       "    <path clip-path=\"url(#pd359cbc70d)\" d=\"M 237.920313 135.255682 \n",
       "L 234.86875 135.290868 \n",
       "L 231.817188 135.321914 \n",
       "L 228.765625 135.348821 \n",
       "L 225.714063 135.371588 \n",
       "L 222.6625 135.390216 \n",
       "L 219.610938 135.404705 \n",
       "L 216.559375 135.415053 \n",
       "L 213.507813 135.421263 \n",
       "L 210.45625 135.423332 \n",
       "L 207.404688 135.421263 \n",
       "L 204.353125 135.415053 \n",
       "L 201.301563 135.404705 \n",
       "L 198.25 135.390216 \n",
       "L 195.198438 135.371588 \n",
       "L 192.146875 135.348821 \n",
       "L 189.095313 135.321914 \n",
       "L 186.04375 135.290868 \n",
       "L 182.992188 135.255682 \n",
       "L 179.940625 135.216357 \n",
       "L 176.889063 135.172892 \n",
       "L 173.8375 135.125288 \n",
       "L 170.785938 135.073544 \n",
       "L 167.734375 135.017661 \n",
       "L 164.682813 134.957638 \n",
       "L 161.63125 134.893475 \n",
       "L 158.579688 134.825174 \n",
       "L 155.528125 134.752732 \n",
       "L 152.476563 134.676151 \n",
       "L 149.425 134.595431 \n",
       "L 146.373438 134.510571 \n",
       "L 143.605741 134.429851 \n",
       "L 143.321875 134.421219 \n",
       "L 140.270313 134.324117 \n",
       "L 137.21875 134.222699 \n",
       "L 134.167188 134.116966 \n",
       "L 131.115625 134.006916 \n",
       "L 128.064063 133.892552 \n",
       "L 125.0125 133.773871 \n",
       "L 121.960938 133.650875 \n",
       "L 118.909375 133.523563 \n",
       "L 115.857813 133.391936 \n",
       "L 112.80625 133.255992 \n",
       "L 109.754688 133.115734 \n",
       "L 106.703125 132.971159 \n",
       "L 103.651563 132.822269 \n",
       "L 100.6 132.669063 \n",
       "L 97.548438 132.511542 \n",
       "L 95.473375 132.401493 \n",
       "L 94.496875 132.347403 \n",
       "L 91.445313 132.173866 \n",
       "L 88.39375 131.995821 \n",
       "L 85.342188 131.813269 \n",
       "L 82.290625 131.626209 \n",
       "L 79.239063 131.434642 \n",
       "L 76.1875 131.238567 \n",
       "L 73.135938 131.037985 \n",
       "L 70.084375 130.832896 \n",
       "L 67.032813 130.623299 \n",
       "L 63.98125 130.409194 \n",
       "L 63.477899 130.373134 \n",
       "L 60.929688 130.182091 \n",
       "L 57.878125 129.948594 \n",
       "L 54.826563 129.71038 \n",
       "L 51.775 129.467449 \n",
       "L 48.723438 129.2198 \n",
       "L 45.671875 128.967435 \n",
       "L 42.620313 128.710352 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pd359cbc70d)\" d=\"M 42.620313 42.788155 \n",
       "L 45.671875 42.531073 \n",
       "L 48.723438 42.278707 \n",
       "L 51.775 42.031059 \n",
       "L 54.826563 41.788127 \n",
       "L 57.878125 41.549913 \n",
       "L 60.929688 41.316416 \n",
       "L 63.477899 41.125373 \n",
       "L 63.98125 41.089313 \n",
       "L 67.032813 40.875209 \n",
       "L 70.084375 40.665612 \n",
       "L 73.135938 40.460522 \n",
       "L 76.1875 40.25994 \n",
       "L 79.239063 40.063866 \n",
       "L 82.290625 39.872299 \n",
       "L 85.342188 39.685239 \n",
       "L 88.39375 39.502687 \n",
       "L 91.445313 39.324642 \n",
       "L 94.496875 39.151104 \n",
       "L 95.473375 39.097015 \n",
       "L 97.548438 38.986966 \n",
       "L 100.6 38.829444 \n",
       "L 103.651563 38.676238 \n",
       "L 106.703125 38.527348 \n",
       "L 109.754688 38.382774 \n",
       "L 112.80625 38.242515 \n",
       "L 115.857813 38.106572 \n",
       "L 118.909375 37.974944 \n",
       "L 121.960938 37.847633 \n",
       "L 125.0125 37.724636 \n",
       "L 128.064063 37.605956 \n",
       "L 131.115625 37.491591 \n",
       "L 134.167188 37.381542 \n",
       "L 137.21875 37.275808 \n",
       "L 140.270313 37.17439 \n",
       "L 143.321875 37.077288 \n",
       "L 143.605741 37.068657 \n",
       "L 146.373438 36.987936 \n",
       "L 149.425 36.903076 \n",
       "L 152.476563 36.822356 \n",
       "L 155.528125 36.745775 \n",
       "L 158.579688 36.673334 \n",
       "L 161.63125 36.605032 \n",
       "L 164.682813 36.54087 \n",
       "L 167.734375 36.480847 \n",
       "L 170.785938 36.424963 \n",
       "L 173.8375 36.37322 \n",
       "L 176.889063 36.325615 \n",
       "L 179.940625 36.28215 \n",
       "L 182.992188 36.242825 \n",
       "L 186.04375 36.207639 \n",
       "L 189.095313 36.176593 \n",
       "L 192.146875 36.149686 \n",
       "L 195.198438 36.126919 \n",
       "L 198.25 36.108291 \n",
       "L 201.301563 36.093803 \n",
       "L 204.353125 36.083454 \n",
       "L 207.404688 36.077245 \n",
       "L 210.45625 36.075175 \n",
       "L 213.507813 36.077245 \n",
       "L 216.559375 36.083454 \n",
       "L 219.610938 36.093803 \n",
       "L 222.6625 36.108291 \n",
       "L 225.714063 36.126919 \n",
       "L 228.765625 36.149686 \n",
       "L 231.817188 36.176593 \n",
       "L 234.86875 36.207639 \n",
       "L 237.920313 36.242825 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_4\">\n",
       "    <path clip-path=\"url(#pd359cbc70d)\" d=\"M 237.920313 142.968527 \n",
       "L 234.86875 142.998775 \n",
       "L 231.817188 143.025463 \n",
       "L 228.765625 143.048594 \n",
       "L 225.714063 143.068166 \n",
       "L 222.6625 143.084179 \n",
       "L 219.610938 143.096634 \n",
       "L 216.559375 143.10553 \n",
       "L 213.507813 143.110868 \n",
       "L 210.45625 143.112647 \n",
       "L 207.404688 143.110868 \n",
       "L 204.353125 143.10553 \n",
       "L 201.301563 143.096634 \n",
       "L 198.25 143.084179 \n",
       "L 195.198438 143.068166 \n",
       "L 192.146875 143.048594 \n",
       "L 189.095313 143.025463 \n",
       "L 186.04375 142.998775 \n",
       "L 182.992188 142.968527 \n",
       "L 179.940625 142.934721 \n",
       "L 176.889063 142.897357 \n",
       "L 173.8375 142.856434 \n",
       "L 170.785938 142.811952 \n",
       "L 167.734375 142.763912 \n",
       "L 164.682813 142.712313 \n",
       "L 161.63125 142.657156 \n",
       "L 158.579688 142.598441 \n",
       "L 155.876875 142.543284 \n",
       "L 155.528125 142.535908 \n",
       "L 152.476563 142.467681 \n",
       "L 149.425 142.395767 \n",
       "L 146.373438 142.320164 \n",
       "L 143.321875 142.240874 \n",
       "L 140.270313 142.157896 \n",
       "L 137.21875 142.071229 \n",
       "L 134.167188 141.980875 \n",
       "L 131.115625 141.886833 \n",
       "L 128.064063 141.789103 \n",
       "L 125.0125 141.687685 \n",
       "L 121.960938 141.582579 \n",
       "L 118.909375 141.473786 \n",
       "L 115.857813 141.361304 \n",
       "L 112.80625 141.245134 \n",
       "L 109.754688 141.125277 \n",
       "L 106.703125 141.001731 \n",
       "L 103.651563 140.874498 \n",
       "L 100.6 140.743577 \n",
       "L 97.548438 140.608967 \n",
       "L 95.473375 140.514925 \n",
       "L 94.496875 140.469 \n",
       "L 91.445313 140.321657 \n",
       "L 88.39375 140.170487 \n",
       "L 85.342188 140.01549 \n",
       "L 82.290625 139.856666 \n",
       "L 79.239063 139.694014 \n",
       "L 76.1875 139.527536 \n",
       "L 73.135938 139.35723 \n",
       "L 70.084375 139.183098 \n",
       "L 67.032813 139.005138 \n",
       "L 63.98125 138.823351 \n",
       "L 60.929688 138.637737 \n",
       "L 58.494602 138.486567 \n",
       "L 57.878125 138.446795 \n",
       "L 54.826563 138.245948 \n",
       "L 51.775 138.041124 \n",
       "L 48.723438 137.832322 \n",
       "L 45.671875 137.619543 \n",
       "L 42.620313 137.402788 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pd359cbc70d)\" d=\"M 42.620313 34.09572 \n",
       "L 45.671875 33.878964 \n",
       "L 48.723438 33.666185 \n",
       "L 51.775 33.457384 \n",
       "L 54.826563 33.252559 \n",
       "L 57.878125 33.051712 \n",
       "L 58.494602 33.01194 \n",
       "L 60.929688 32.86077 \n",
       "L 63.98125 32.675156 \n",
       "L 67.032813 32.493369 \n",
       "L 70.084375 32.31541 \n",
       "L 73.135938 32.141277 \n",
       "L 76.1875 31.970972 \n",
       "L 79.239063 31.804493 \n",
       "L 82.290625 31.641842 \n",
       "L 85.342188 31.483017 \n",
       "L 88.39375 31.32802 \n",
       "L 91.445313 31.17685 \n",
       "L 94.496875 31.029507 \n",
       "L 95.473375 30.983582 \n",
       "L 97.548438 30.88954 \n",
       "L 100.6 30.754931 \n",
       "L 103.651563 30.624009 \n",
       "L 106.703125 30.496776 \n",
       "L 109.754688 30.373231 \n",
       "L 112.80625 30.253373 \n",
       "L 115.857813 30.137204 \n",
       "L 118.909375 30.024722 \n",
       "L 121.960938 29.915928 \n",
       "L 125.0125 29.810822 \n",
       "L 128.064063 29.709404 \n",
       "L 131.115625 29.611674 \n",
       "L 134.167188 29.517632 \n",
       "L 137.21875 29.427278 \n",
       "L 140.270313 29.340612 \n",
       "L 143.321875 29.257634 \n",
       "L 146.373438 29.178343 \n",
       "L 149.425 29.102741 \n",
       "L 152.476563 29.030826 \n",
       "L 155.528125 28.9626 \n",
       "L 155.876875 28.955224 \n",
       "L 158.579688 28.900067 \n",
       "L 161.63125 28.841351 \n",
       "L 164.682813 28.786194 \n",
       "L 167.734375 28.734595 \n",
       "L 170.785938 28.686555 \n",
       "L 173.8375 28.642074 \n",
       "L 176.889063 28.601151 \n",
       "L 179.940625 28.563786 \n",
       "L 182.992188 28.52998 \n",
       "L 186.04375 28.499733 \n",
       "L 189.095313 28.473044 \n",
       "L 192.146875 28.449914 \n",
       "L 195.198438 28.430342 \n",
       "L 198.25 28.414328 \n",
       "L 201.301563 28.401874 \n",
       "L 204.353125 28.392977 \n",
       "L 207.404688 28.387639 \n",
       "L 210.45625 28.38586 \n",
       "L 213.507813 28.387639 \n",
       "L 216.559375 28.392977 \n",
       "L 219.610938 28.401874 \n",
       "L 222.6625 28.414328 \n",
       "L 225.714063 28.430342 \n",
       "L 228.765625 28.449914 \n",
       "L 231.817188 28.473044 \n",
       "L 234.86875 28.499733 \n",
       "L 237.920313 28.52998 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_5\">\n",
       "    <path clip-path=\"url(#pd359cbc70d)\" d=\"M 73.993118 146.6 \n",
       "L 73.135938 146.557026 \n",
       "L 70.084375 146.400602 \n",
       "L 67.032813 146.24074 \n",
       "L 63.98125 146.07744 \n",
       "L 60.929688 145.910702 \n",
       "L 57.878125 145.740526 \n",
       "L 54.826563 145.566912 \n",
       "L 51.775 145.389861 \n",
       "L 48.723438 145.209371 \n",
       "L 45.671875 145.025444 \n",
       "L 42.620313 144.838079 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "    <path clip-path=\"url(#pd359cbc70d)\" d=\"M 42.620313 26.660429 \n",
       "L 45.671875 26.473063 \n",
       "L 48.723438 26.289136 \n",
       "L 51.775 26.108647 \n",
       "L 54.826563 25.931595 \n",
       "L 57.878125 25.757981 \n",
       "L 60.929688 25.587805 \n",
       "L 63.98125 25.421068 \n",
       "L 67.032813 25.257768 \n",
       "L 70.084375 25.097905 \n",
       "L 73.135938 24.941481 \n",
       "L 73.993118 24.898507 \n",
       "L 76.1875 24.792102 \n",
       "L 79.239063 24.647457 \n",
       "L 82.290625 24.506137 \n",
       "L 85.342188 24.368142 \n",
       "L 88.39375 24.233472 \n",
       "L 91.445313 24.102127 \n",
       "L 94.496875 23.974108 \n",
       "L 97.548438 23.849414 \n",
       "L 100.6 23.728045 \n",
       "L 103.651563 23.610001 \n",
       "L 106.703125 23.495283 \n",
       "L 109.754688 23.383889 \n",
       "L 112.80625 23.275821 \n",
       "L 115.857813 23.171078 \n",
       "L 118.909375 23.06966 \n",
       "L 121.960938 22.971567 \n",
       "L 125.0125 22.8768 \n",
       "L 125.234432 22.870149 \n",
       "L 128.064063 22.788049 \n",
       "L 131.115625 22.702729 \n",
       "L 134.167188 22.620629 \n",
       "L 137.21875 22.541748 \n",
       "L 140.270313 22.466087 \n",
       "L 143.321875 22.393646 \n",
       "L 146.373438 22.324424 \n",
       "L 149.425 22.258422 \n",
       "L 152.476563 22.19564 \n",
       "L 155.528125 22.136077 \n",
       "L 158.579688 22.079733 \n",
       "L 161.63125 22.02661 \n",
       "L 164.682813 21.976706 \n",
       "L 167.734375 21.930021 \n",
       "L 170.785938 21.886557 \n",
       "L 173.8375 21.846311 \n",
       "L 176.889063 21.809286 \n",
       "L 179.940625 21.77548 \n",
       "L 182.992188 21.744893 \n",
       "L 186.04375 21.717527 \n",
       "L 189.095313 21.69338 \n",
       "L 192.146875 21.672452 \n",
       "L 195.198438 21.654744 \n",
       "L 198.25 21.640256 \n",
       "L 201.301563 21.628987 \n",
       "L 204.353125 21.620938 \n",
       "L 207.404688 21.616109 \n",
       "L 210.45625 21.614499 \n",
       "L 213.507813 21.616109 \n",
       "L 216.559375 21.620938 \n",
       "L 219.610938 21.628987 \n",
       "L 222.6625 21.640256 \n",
       "L 225.714063 21.654744 \n",
       "L 228.765625 21.672452 \n",
       "L 231.817188 21.69338 \n",
       "L 234.86875 21.717527 \n",
       "L 237.920313 21.744893 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_6\">\n",
       "    <path clip-path=\"url(#pd359cbc70d)\" d=\"M 42.620313 20.069455 \n",
       "L 45.671875 19.899385 \n",
       "L 48.723438 19.732435 \n",
       "L 51.775 19.568606 \n",
       "L 54.826563 19.407898 \n",
       "L 57.878125 19.25031 \n",
       "L 60.929688 19.095843 \n",
       "L 63.98125 18.944496 \n",
       "L 66.679474 18.813433 \n",
       "L 67.032813 18.796782 \n",
       "L 70.084375 18.656008 \n",
       "L 73.135938 18.518261 \n",
       "L 76.1875 18.383542 \n",
       "L 79.239063 18.25185 \n",
       "L 82.290625 18.123186 \n",
       "L 85.342188 17.997548 \n",
       "L 88.39375 17.874939 \n",
       "L 91.445313 17.755356 \n",
       "L 94.496875 17.638802 \n",
       "L 97.548438 17.525274 \n",
       "L 100.6 17.414774 \n",
       "L 103.651563 17.307301 \n",
       "L 106.703125 17.202856 \n",
       "L 109.754688 17.101438 \n",
       "L 112.80625 17.003047 \n",
       "L 115.857813 16.907684 \n",
       "L 118.909375 16.815349 \n",
       "L 119.943803 16.785075 \n",
       "L 121.960938 16.727751 \n",
       "L 125.0125 16.643971 \n",
       "L 128.064063 16.563131 \n",
       "L 131.115625 16.48523 \n",
       "L 134.167188 16.410269 \n",
       "L 137.21875 16.338248 \n",
       "L 140.270313 16.269166 \n",
       "L 143.321875 16.203024 \n",
       "L 146.373438 16.139822 \n",
       "L 149.425 16.079559 \n",
       "L 152.476563 16.022236 \n",
       "L 155.528125 15.967852 \n",
       "L 158.579688 15.916408 \n",
       "L 161.63125 15.867904 \n",
       "L 164.682813 15.822339 \n",
       "L 167.734375 15.779714 \n",
       "L 170.785938 15.740029 \n",
       "L 173.8375 15.703284 \n",
       "L 176.889063 15.669478 \n",
       "L 179.940625 15.638611 \n",
       "L 182.992188 15.610685 \n",
       "L 186.04375 15.585698 \n",
       "L 189.095313 15.56365 \n",
       "L 192.146875 15.544543 \n",
       "L 195.198438 15.528374 \n",
       "L 198.25 15.515146 \n",
       "L 201.301563 15.504857 \n",
       "L 204.353125 15.497508 \n",
       "L 207.404688 15.493099 \n",
       "L 210.45625 15.491629 \n",
       "L 213.507813 15.493099 \n",
       "L 216.559375 15.497508 \n",
       "L 219.610938 15.504857 \n",
       "L 222.6625 15.515146 \n",
       "L 225.714063 15.528374 \n",
       "L 228.765625 15.544543 \n",
       "L 231.817188 15.56365 \n",
       "L 234.86875 15.585698 \n",
       "L 237.920313 15.610685 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_7\">\n",
       "    <path clip-path=\"url(#pd359cbc70d)\" d=\"M 42.620313 14.078216 \n",
       "L 45.671875 13.922518 \n",
       "L 48.723438 13.769677 \n",
       "L 51.775 13.619693 \n",
       "L 54.826563 13.472566 \n",
       "L 57.878125 13.328295 \n",
       "L 60.929688 13.186881 \n",
       "L 63.98125 13.048325 \n",
       "L 67.032813 12.912625 \n",
       "L 70.084375 12.779781 \n",
       "L 71.291587 12.728358 \n",
       "L 73.135938 12.651947 \n",
       "L 76.1875 12.528301 \n",
       "L 79.239063 12.407433 \n",
       "L 82.290625 12.289344 \n",
       "L 85.342188 12.174033 \n",
       "L 88.39375 12.061501 \n",
       "L 91.445313 11.951747 \n",
       "L 94.496875 11.844772 \n",
       "L 97.548438 11.740576 \n",
       "L 100.6 11.639158 \n",
       "L 103.651563 11.540518 \n",
       "L 106.703125 11.444658 \n",
       "L 109.754688 11.351575 \n",
       "L 112.80625 11.261272 \n",
       "L 115.857813 11.173747 \n",
       "L 118.909375 11.089 \n",
       "L 121.960938 11.007032 \n",
       "L 125.0125 10.927843 \n",
       "L 128.064063 10.851432 \n",
       "L 131.115625 10.7778 \n",
       "L 134.167188 10.706946 \n",
       "L 134.478571 10.7 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_7\">\n",
       "    <path clip-path=\"url(#pd359cbc70d)\" d=\"M 57.878125 126.316418 \n",
       "L 76.1875 28.955224 \n",
       "L 101.454438 116.580299 \n",
       "L 127.168124 86.398328 \n",
       "L 150.019542 69.749564 \n",
       "L 168.697656 99.824437 \n",
       "L 183.047744 81.081434 \n",
       "L 193.511809 82.9127 \n",
       "L 200.777175 90.636062 \n",
       "L 205.571346 82.769403 \n",
       "L 208.554621 85.987715 \n",
       "L 210.274453 87.024564 \n",
       "L 211.156185 84.482244 \n",
       "L 211.513059 86.251907 \n",
       "L 211.564679 85.93037 \n",
       "L 211.457477 85.334922 \n",
       "L 211.283729 86.031594 \n",
       "L 211.097558 85.702313 \n",
       "L 210.927515 85.65033 \n",
       "L 210.785942 85.861755 \n",
       "L 210.675592 85.697464 \n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    <defs>\n",
       "     <path d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" id=\"m9c4f471e71\" style=\"stroke:#ff7f0e;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#pd359cbc70d)\">\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"57.878125\" xlink:href=\"#m9c4f471e71\" y=\"126.316418\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"76.1875\" xlink:href=\"#m9c4f471e71\" y=\"28.955224\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"101.454438\" xlink:href=\"#m9c4f471e71\" y=\"116.580299\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"127.168124\" xlink:href=\"#m9c4f471e71\" y=\"86.398328\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"150.019542\" xlink:href=\"#m9c4f471e71\" y=\"69.749564\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"168.697656\" xlink:href=\"#m9c4f471e71\" y=\"99.824437\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"183.047744\" xlink:href=\"#m9c4f471e71\" y=\"81.081434\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"193.511809\" xlink:href=\"#m9c4f471e71\" y=\"82.9127\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"200.777175\" xlink:href=\"#m9c4f471e71\" y=\"90.636062\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"205.571346\" xlink:href=\"#m9c4f471e71\" y=\"82.769403\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"208.554621\" xlink:href=\"#m9c4f471e71\" y=\"85.987715\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"210.274453\" xlink:href=\"#m9c4f471e71\" y=\"87.024564\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"211.156185\" xlink:href=\"#m9c4f471e71\" y=\"84.482244\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"211.513059\" xlink:href=\"#m9c4f471e71\" y=\"86.251907\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"211.564679\" xlink:href=\"#m9c4f471e71\" y=\"85.93037\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"211.457477\" xlink:href=\"#m9c4f471e71\" y=\"85.334922\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"211.283729\" xlink:href=\"#m9c4f471e71\" y=\"86.031594\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"211.097558\" xlink:href=\"#m9c4f471e71\" y=\"85.702313\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"210.927515\" xlink:href=\"#m9c4f471e71\" y=\"85.65033\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"210.785942\" xlink:href=\"#m9c4f471e71\" y=\"85.861755\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#ff7f0e;\" x=\"210.675592\" xlink:href=\"#m9c4f471e71\" y=\"85.697464\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 42.620312 146.6 \n",
       "L 42.620312 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 237.920313 146.6 \n",
       "L 237.920313 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 42.620313 146.6 \n",
       "L 237.920313 146.6 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 42.620313 10.7 \n",
       "L 237.920313 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pd359cbc70d\">\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"42.620312\" y=\"10.7\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "eta = 0.6\n",
    "d2lt.show_trace_2d(f_2d, d2lt.train_2d(momentum_2d))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 指数加权移动平均\n",
    "\n",
    "为了从数学上理解动量法，让我们先解释一下指数加权移动平均（exponentially weighted moving average）。给定超参数$0 \\leq \\gamma < 1$，当前时间步$t$的变量$y_t$是上一时间步$t-1$的变量$y_{t-1}$和当前时间步另一变量$x_t$的线性组合：\n",
    "\n",
    "$$y_t = \\gamma y_{t-1} + (1-\\gamma) x_t.$$\n",
    "\n",
    "我们可以对$y_t$展开：\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "y_t  &= (1-\\gamma) x_t + \\gamma y_{t-1}\\\\\n",
    "         &= (1-\\gamma)x_t + (1-\\gamma) \\cdot \\gamma x_{t-1} + \\gamma^2y_{t-2}\\\\\n",
    "         &= (1-\\gamma)x_t + (1-\\gamma) \\cdot \\gamma x_{t-1} + (1-\\gamma) \\cdot \\gamma^2x_{t-2} + \\gamma^3y_{t-3}\\\\\n",
    "         &\\ldots\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "令$n = 1/(1-\\gamma)$，那么 $\\left(1-1/n\\right)^n = \\gamma^{1/(1-\\gamma)}$。因为\n",
    "\n",
    "$$ \\lim_{n \\rightarrow \\infty}  \\left(1-\\frac{1}{n}\\right)^n = \\exp(-1) \\approx 0.3679,$$\n",
    "\n",
    "所以当$\\gamma \\rightarrow 1$时，$\\gamma^{1/(1-\\gamma)}=\\exp(-1)$，如$0.95^{20} \\approx \\exp(-1)$。如果把$\\exp(-1)$当作一个比较小的数，我们可以在近似中忽略所有含$\\gamma^{1/(1-\\gamma)}$和比$\\gamma^{1/(1-\\gamma)}$更高阶的系数的项。例如，当$\\gamma=0.95$时，\n",
    "\n",
    "$$y_t \\approx 0.05 \\sum_{i=0}^{19} 0.95^i x_{t-i}.$$\n",
    "\n",
    "因此，在实际中，我们常常将$y_t$看作是对最近$1/(1-\\gamma)$个时间步的$x_t$值的加权平均。例如，当$\\gamma = 0.95$时，$y_t$可以被看作对最近20个时间步的$x_t$值的加权平均；当$\\gamma = 0.9$时，$y_t$可以看作是对最近10个时间步的$x_t$值的加权平均。而且，离当前时间步$t$越近的$x_t$值获得的权重越大（越接近1）。\n",
    "\n",
    "\n",
    "### 由指数加权移动平均理解动量法\n",
    "\n",
    "现在，我们对动量法的速度变量做变形：\n",
    "\n",
    "$$\\boldsymbol{v}_t \\leftarrow \\gamma \\boldsymbol{v}_{t-1} + (1 - \\gamma) \\left(\\frac{\\eta_t}{1 - \\gamma} \\boldsymbol{g}_t\\right). $$\n",
    "\n",
    "由指数加权移动平均的形式可得，速度变量$\\boldsymbol{v}_t$实际上对序列$\\{\\eta_{t-i}\\boldsymbol{g}_{t-i} /(1-\\gamma):i=0,\\ldots,1/(1-\\gamma)-1\\}$做了指数加权移动平均。换句话说，相比于小批量随机梯度下降，动量法在每个时间步的自变量更新量近似于将前者对应的最近$1/(1-\\gamma)$个时间步的更新量做了指数加权移动平均后再除以$1-\\gamma$。所以，在动量法中，自变量在各个方向上的移动幅度不仅取决当前梯度，还取决于过去的各个梯度在各个方向上是否一致。在本节之前示例的优化问题中，所有梯度在水平方向上为正（向右），而在竖直方向上时正（向上）时负（向下）。这样，我们就可以使用较大的学习率，从而使自变量向最优解更快移动。\n",
    "\n",
    "\n",
    "## 从零开始实现\n",
    "\n",
    "相对于小批量随机梯度下降，动量法需要对每一个自变量维护一个同它一样形状的速度变量，且超参数里多了动量超参数。实现中，我们将速度变量用更广义的状态变量`states`表示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "13"
    }
   },
   "outputs": [],
   "source": [
    "features, labels = d2lt.get_data_ch7()\n",
    "\n",
    "def init_momentum_states():\n",
    "    v_w = torch.zeros(features.shape[1], 1)\n",
    "    v_b = torch.zeros(1)\n",
    "    return (v_w, v_b)\n",
    "\n",
    "def sgd_momentum(params, states, hyperparams):\n",
    "    for p, v in zip(params, states):\n",
    "        v.data = hyperparams['momentum'] * v.data + hyperparams['lr'] * p.grad.data\n",
    "        p.data -= v.data\n",
    "        p.grad.data.zero_()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们先将动量超参数`momentum`设0.5，这时可以看成是特殊的小批量随机梯度下降：其小批量随机梯度为最近2个时间步的2倍小批量梯度的加权平均。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "15"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 0.243377, 0.080929 sec per epoch\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 256.14375 184.15625\" width=\"256.14375pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 184.15625 \n",
       "L 256.14375 184.15625 \n",
       "L 256.14375 -0 \n",
       "L 0 -0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 50.14375 146.6 \n",
       "L 245.44375 146.6 \n",
       "L 245.44375 10.7 \n",
       "L 50.14375 10.7 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"m9fbb41b9bb\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"59.021023\" xlink:href=\"#m9fbb41b9bb\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0.0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-30\"/>\n",
       "       <path d=\"M 10.6875 12.40625 \n",
       "L 21 12.40625 \n",
       "L 21 0 \n",
       "L 10.6875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-2e\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(51.06946 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"103.407386\" xlink:href=\"#m9fbb41b9bb\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 0.5 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"DejaVuSans-35\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(95.455824 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"147.79375\" xlink:href=\"#m9fbb41b9bb\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 1.0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-31\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(139.842187 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"192.180114\" xlink:href=\"#m9fbb41b9bb\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 1.5 -->\n",
       "      <g transform=\"translate(184.228551 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"236.566477\" xlink:href=\"#m9fbb41b9bb\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 2.0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-32\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(228.614915 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <defs>\n",
       "      <path d=\"M 56.203125 29.59375 \n",
       "L 56.203125 25.203125 \n",
       "L 14.890625 25.203125 \n",
       "Q 15.484375 15.921875 20.484375 11.0625 \n",
       "Q 25.484375 6.203125 34.421875 6.203125 \n",
       "Q 39.59375 6.203125 44.453125 7.46875 \n",
       "Q 49.3125 8.734375 54.109375 11.28125 \n",
       "L 54.109375 2.78125 \n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \n",
       "Q 5.515625 13.8125 5.515625 26.8125 \n",
       "Q 5.515625 40.234375 12.765625 48.109375 \n",
       "Q 20.015625 56 32.328125 56 \n",
       "Q 43.359375 56 49.78125 48.890625 \n",
       "Q 56.203125 41.796875 56.203125 29.59375 \n",
       "z\n",
       "M 47.21875 32.234375 \n",
       "Q 47.125 39.59375 43.09375 43.984375 \n",
       "Q 39.0625 48.390625 32.421875 48.390625 \n",
       "Q 24.90625 48.390625 20.390625 44.140625 \n",
       "Q 15.875 39.890625 15.1875 32.171875 \n",
       "z\n",
       "\" id=\"DejaVuSans-65\"/>\n",
       "      <path d=\"M 18.109375 8.203125 \n",
       "L 18.109375 -20.796875 \n",
       "L 9.078125 -20.796875 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.390625 \n",
       "Q 20.953125 51.265625 25.265625 53.625 \n",
       "Q 29.59375 56 35.59375 56 \n",
       "Q 45.5625 56 51.78125 48.09375 \n",
       "Q 58.015625 40.1875 58.015625 27.296875 \n",
       "Q 58.015625 14.40625 51.78125 6.484375 \n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \n",
       "Q 20.953125 3.328125 18.109375 8.203125 \n",
       "z\n",
       "M 48.6875 27.296875 \n",
       "Q 48.6875 37.203125 44.609375 42.84375 \n",
       "Q 40.53125 48.484375 33.40625 48.484375 \n",
       "Q 26.265625 48.484375 22.1875 42.84375 \n",
       "Q 18.109375 37.203125 18.109375 27.296875 \n",
       "Q 18.109375 17.390625 22.1875 11.75 \n",
       "Q 26.265625 6.109375 33.40625 6.109375 \n",
       "Q 40.53125 6.109375 44.609375 11.75 \n",
       "Q 48.6875 17.390625 48.6875 27.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-70\"/>\n",
       "      <path d=\"M 30.609375 48.390625 \n",
       "Q 23.390625 48.390625 19.1875 42.75 \n",
       "Q 14.984375 37.109375 14.984375 27.296875 \n",
       "Q 14.984375 17.484375 19.15625 11.84375 \n",
       "Q 23.34375 6.203125 30.609375 6.203125 \n",
       "Q 37.796875 6.203125 41.984375 11.859375 \n",
       "Q 46.1875 17.53125 46.1875 27.296875 \n",
       "Q 46.1875 37.015625 41.984375 42.703125 \n",
       "Q 37.796875 48.390625 30.609375 48.390625 \n",
       "z\n",
       "M 30.609375 56 \n",
       "Q 42.328125 56 49.015625 48.375 \n",
       "Q 55.71875 40.765625 55.71875 27.296875 \n",
       "Q 55.71875 13.875 49.015625 6.21875 \n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \n",
       "Q 5.515625 13.875 5.515625 27.296875 \n",
       "Q 5.515625 40.765625 12.171875 48.375 \n",
       "Q 18.84375 56 30.609375 56 \n",
       "z\n",
       "\" id=\"DejaVuSans-6f\"/>\n",
       "      <path d=\"M 48.78125 52.59375 \n",
       "L 48.78125 44.1875 \n",
       "Q 44.96875 46.296875 41.140625 47.34375 \n",
       "Q 37.3125 48.390625 33.40625 48.390625 \n",
       "Q 24.65625 48.390625 19.8125 42.84375 \n",
       "Q 14.984375 37.3125 14.984375 27.296875 \n",
       "Q 14.984375 17.28125 19.8125 11.734375 \n",
       "Q 24.65625 6.203125 33.40625 6.203125 \n",
       "Q 37.3125 6.203125 41.140625 7.25 \n",
       "Q 44.96875 8.296875 48.78125 10.40625 \n",
       "L 48.78125 2.09375 \n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \n",
       "Q 5.515625 14.109375 5.515625 27.296875 \n",
       "Q 5.515625 40.671875 12.859375 48.328125 \n",
       "Q 20.21875 56 33.015625 56 \n",
       "Q 37.15625 56 41.109375 55.140625 \n",
       "Q 45.0625 54.296875 48.78125 52.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-63\"/>\n",
       "      <path d=\"M 54.890625 33.015625 \n",
       "L 54.890625 0 \n",
       "L 45.90625 0 \n",
       "L 45.90625 32.71875 \n",
       "Q 45.90625 40.484375 42.875 44.328125 \n",
       "Q 39.84375 48.1875 33.796875 48.1875 \n",
       "Q 26.515625 48.1875 22.3125 43.546875 \n",
       "Q 18.109375 38.921875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 75.984375 \n",
       "L 18.109375 75.984375 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.34375 51.125 25.703125 53.5625 \n",
       "Q 30.078125 56 35.796875 56 \n",
       "Q 45.21875 56 50.046875 50.171875 \n",
       "Q 54.890625 44.34375 54.890625 33.015625 \n",
       "z\n",
       "\" id=\"DejaVuSans-68\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(132.565625 174.876562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"m0d2686a712\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m0d2686a712\" y=\"136.905993\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0.25 -->\n",
       "      <g transform=\"translate(20.878125 140.705212)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m0d2686a712\" y=\"112.827048\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0.30 -->\n",
       "      <defs>\n",
       "       <path d=\"M 40.578125 39.3125 \n",
       "Q 47.65625 37.796875 51.625 33 \n",
       "Q 55.609375 28.21875 55.609375 21.1875 \n",
       "Q 55.609375 10.40625 48.1875 4.484375 \n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \n",
       "Q 12.796875 0.390625 7.625 2.203125 \n",
       "L 7.625 11.71875 \n",
       "Q 11.71875 9.328125 16.59375 8.109375 \n",
       "Q 21.484375 6.890625 26.8125 6.890625 \n",
       "Q 36.078125 6.890625 40.9375 10.546875 \n",
       "Q 45.796875 14.203125 45.796875 21.1875 \n",
       "Q 45.796875 27.640625 41.28125 31.265625 \n",
       "Q 36.765625 34.90625 28.71875 34.90625 \n",
       "L 20.21875 34.90625 \n",
       "L 20.21875 43.015625 \n",
       "L 29.109375 43.015625 \n",
       "Q 36.375 43.015625 40.234375 45.921875 \n",
       "Q 44.09375 48.828125 44.09375 54.296875 \n",
       "Q 44.09375 59.90625 40.109375 62.90625 \n",
       "Q 36.140625 65.921875 28.71875 65.921875 \n",
       "Q 24.65625 65.921875 20.015625 65.03125 \n",
       "Q 15.375 64.15625 9.8125 62.3125 \n",
       "L 9.8125 71.09375 \n",
       "Q 15.4375 72.65625 20.34375 73.4375 \n",
       "Q 25.25 74.21875 29.59375 74.21875 \n",
       "Q 40.828125 74.21875 47.359375 69.109375 \n",
       "Q 53.90625 64.015625 53.90625 55.328125 \n",
       "Q 53.90625 49.265625 50.4375 45.09375 \n",
       "Q 46.96875 40.921875 40.578125 39.3125 \n",
       "z\n",
       "\" id=\"DejaVuSans-33\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(20.878125 116.626266)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m0d2686a712\" y=\"88.748102\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0.35 -->\n",
       "      <g transform=\"translate(20.878125 92.54732)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m0d2686a712\" y=\"64.669156\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.40 -->\n",
       "      <defs>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"DejaVuSans-34\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(20.878125 68.468374)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m0d2686a712\" y=\"40.59021\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.45 -->\n",
       "      <g transform=\"translate(20.878125 44.389428)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m0d2686a712\" y=\"16.511264\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 0.50 -->\n",
       "      <g transform=\"translate(20.878125 20.310482)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- loss -->\n",
       "     <defs>\n",
       "      <path d=\"M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-6c\"/>\n",
       "      <path d=\"M 44.28125 53.078125 \n",
       "L 44.28125 44.578125 \n",
       "Q 40.484375 46.53125 36.375 47.5 \n",
       "Q 32.28125 48.484375 27.875 48.484375 \n",
       "Q 21.1875 48.484375 17.84375 46.4375 \n",
       "Q 14.5 44.390625 14.5 40.28125 \n",
       "Q 14.5 37.15625 16.890625 35.375 \n",
       "Q 19.28125 33.59375 26.515625 31.984375 \n",
       "L 29.59375 31.296875 \n",
       "Q 39.15625 29.25 43.1875 25.515625 \n",
       "Q 47.21875 21.78125 47.21875 15.09375 \n",
       "Q 47.21875 7.46875 41.1875 3.015625 \n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \n",
       "Q 10.6875 0.296875 5.421875 2 \n",
       "L 5.421875 11.28125 \n",
       "Q 10.40625 8.6875 15.234375 7.390625 \n",
       "Q 20.0625 6.109375 24.8125 6.109375 \n",
       "Q 31.15625 6.109375 34.5625 8.28125 \n",
       "Q 37.984375 10.453125 37.984375 14.40625 \n",
       "Q 37.984375 18.0625 35.515625 20.015625 \n",
       "Q 33.0625 21.96875 24.703125 23.78125 \n",
       "L 21.578125 24.515625 \n",
       "Q 13.234375 26.265625 9.515625 29.90625 \n",
       "Q 5.8125 33.546875 5.8125 39.890625 \n",
       "Q 5.8125 47.609375 11.28125 51.796875 \n",
       "Q 16.75 56 26.8125 56 \n",
       "Q 31.78125 56 36.171875 55.265625 \n",
       "Q 40.578125 54.546875 44.28125 53.078125 \n",
       "z\n",
       "\" id=\"DejaVuSans-73\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(14.798437 88.307812)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-6c\"/>\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_12\">\n",
       "    <path clip-path=\"url(#pedf892093f)\" d=\"M 59.021023 16.877273 \n",
       "L 64.939205 65.886533 \n",
       "L 70.857386 93.685007 \n",
       "L 76.775568 107.267729 \n",
       "L 82.69375 122.515015 \n",
       "L 88.611932 130.315491 \n",
       "L 94.530114 136.458335 \n",
       "L 100.448295 137.632292 \n",
       "L 106.366477 139.085141 \n",
       "L 112.284659 139.556459 \n",
       "L 118.202841 139.253384 \n",
       "L 124.121023 139.151441 \n",
       "L 130.039205 137.082281 \n",
       "L 135.957386 137.754106 \n",
       "L 141.875568 137.400505 \n",
       "L 147.79375 139.921987 \n",
       "L 153.711932 139.501447 \n",
       "L 159.630114 138.843005 \n",
       "L 165.548295 139.267126 \n",
       "L 171.466477 139.722112 \n",
       "L 177.384659 139.914316 \n",
       "L 183.302841 138.413309 \n",
       "L 189.221023 137.775096 \n",
       "L 195.139205 137.806269 \n",
       "L 201.057386 140.060342 \n",
       "L 206.975568 140.422727 \n",
       "L 212.89375 139.866014 \n",
       "L 218.811932 140.025997 \n",
       "L 224.730114 139.05226 \n",
       "L 230.648295 139.77027 \n",
       "L 236.566477 140.095591 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 50.14375 146.6 \n",
       "L 50.14375 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 245.44375 146.6 \n",
       "L 245.44375 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 50.14375 146.6 \n",
       "L 245.44375 146.6 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 50.14375 10.7 \n",
       "L 245.44375 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pedf892093f\">\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"50.14375\" y=\"10.7\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "d2lt.train_ch7(sgd_momentum, init_momentum_states(),\n",
    "              {'lr': 0.02, 'momentum': 0.5}, features, labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将动量超参数`momentum`增大到0.9，这时依然可以看成是特殊的小批量随机梯度下降：其小批量随机梯度为最近10个时间步的10倍小批量梯度的加权平均。我们先保持学习率0.02不变。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "8"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 0.250928, 0.088715 sec per epoch\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 256.14375 184.15625\" width=\"256.14375pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 184.15625 \n",
       "L 256.14375 184.15625 \n",
       "L 256.14375 -0 \n",
       "L 0 -0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 50.14375 146.6 \n",
       "L 245.44375 146.6 \n",
       "L 245.44375 10.7 \n",
       "L 50.14375 10.7 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"mcac89245ab\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"59.021023\" xlink:href=\"#mcac89245ab\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0.0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-30\"/>\n",
       "       <path d=\"M 10.6875 12.40625 \n",
       "L 21 12.40625 \n",
       "L 21 0 \n",
       "L 10.6875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-2e\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(51.06946 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"103.407386\" xlink:href=\"#mcac89245ab\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 0.5 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"DejaVuSans-35\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(95.455824 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"147.79375\" xlink:href=\"#mcac89245ab\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 1.0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-31\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(139.842187 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"192.180114\" xlink:href=\"#mcac89245ab\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 1.5 -->\n",
       "      <g transform=\"translate(184.228551 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"236.566477\" xlink:href=\"#mcac89245ab\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 2.0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-32\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(228.614915 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <defs>\n",
       "      <path d=\"M 56.203125 29.59375 \n",
       "L 56.203125 25.203125 \n",
       "L 14.890625 25.203125 \n",
       "Q 15.484375 15.921875 20.484375 11.0625 \n",
       "Q 25.484375 6.203125 34.421875 6.203125 \n",
       "Q 39.59375 6.203125 44.453125 7.46875 \n",
       "Q 49.3125 8.734375 54.109375 11.28125 \n",
       "L 54.109375 2.78125 \n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \n",
       "Q 5.515625 13.8125 5.515625 26.8125 \n",
       "Q 5.515625 40.234375 12.765625 48.109375 \n",
       "Q 20.015625 56 32.328125 56 \n",
       "Q 43.359375 56 49.78125 48.890625 \n",
       "Q 56.203125 41.796875 56.203125 29.59375 \n",
       "z\n",
       "M 47.21875 32.234375 \n",
       "Q 47.125 39.59375 43.09375 43.984375 \n",
       "Q 39.0625 48.390625 32.421875 48.390625 \n",
       "Q 24.90625 48.390625 20.390625 44.140625 \n",
       "Q 15.875 39.890625 15.1875 32.171875 \n",
       "z\n",
       "\" id=\"DejaVuSans-65\"/>\n",
       "      <path d=\"M 18.109375 8.203125 \n",
       "L 18.109375 -20.796875 \n",
       "L 9.078125 -20.796875 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.390625 \n",
       "Q 20.953125 51.265625 25.265625 53.625 \n",
       "Q 29.59375 56 35.59375 56 \n",
       "Q 45.5625 56 51.78125 48.09375 \n",
       "Q 58.015625 40.1875 58.015625 27.296875 \n",
       "Q 58.015625 14.40625 51.78125 6.484375 \n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \n",
       "Q 20.953125 3.328125 18.109375 8.203125 \n",
       "z\n",
       "M 48.6875 27.296875 \n",
       "Q 48.6875 37.203125 44.609375 42.84375 \n",
       "Q 40.53125 48.484375 33.40625 48.484375 \n",
       "Q 26.265625 48.484375 22.1875 42.84375 \n",
       "Q 18.109375 37.203125 18.109375 27.296875 \n",
       "Q 18.109375 17.390625 22.1875 11.75 \n",
       "Q 26.265625 6.109375 33.40625 6.109375 \n",
       "Q 40.53125 6.109375 44.609375 11.75 \n",
       "Q 48.6875 17.390625 48.6875 27.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-70\"/>\n",
       "      <path d=\"M 30.609375 48.390625 \n",
       "Q 23.390625 48.390625 19.1875 42.75 \n",
       "Q 14.984375 37.109375 14.984375 27.296875 \n",
       "Q 14.984375 17.484375 19.15625 11.84375 \n",
       "Q 23.34375 6.203125 30.609375 6.203125 \n",
       "Q 37.796875 6.203125 41.984375 11.859375 \n",
       "Q 46.1875 17.53125 46.1875 27.296875 \n",
       "Q 46.1875 37.015625 41.984375 42.703125 \n",
       "Q 37.796875 48.390625 30.609375 48.390625 \n",
       "z\n",
       "M 30.609375 56 \n",
       "Q 42.328125 56 49.015625 48.375 \n",
       "Q 55.71875 40.765625 55.71875 27.296875 \n",
       "Q 55.71875 13.875 49.015625 6.21875 \n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \n",
       "Q 5.515625 13.875 5.515625 27.296875 \n",
       "Q 5.515625 40.765625 12.171875 48.375 \n",
       "Q 18.84375 56 30.609375 56 \n",
       "z\n",
       "\" id=\"DejaVuSans-6f\"/>\n",
       "      <path d=\"M 48.78125 52.59375 \n",
       "L 48.78125 44.1875 \n",
       "Q 44.96875 46.296875 41.140625 47.34375 \n",
       "Q 37.3125 48.390625 33.40625 48.390625 \n",
       "Q 24.65625 48.390625 19.8125 42.84375 \n",
       "Q 14.984375 37.3125 14.984375 27.296875 \n",
       "Q 14.984375 17.28125 19.8125 11.734375 \n",
       "Q 24.65625 6.203125 33.40625 6.203125 \n",
       "Q 37.3125 6.203125 41.140625 7.25 \n",
       "Q 44.96875 8.296875 48.78125 10.40625 \n",
       "L 48.78125 2.09375 \n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \n",
       "Q 5.515625 14.109375 5.515625 27.296875 \n",
       "Q 5.515625 40.671875 12.859375 48.328125 \n",
       "Q 20.21875 56 33.015625 56 \n",
       "Q 37.15625 56 41.109375 55.140625 \n",
       "Q 45.0625 54.296875 48.78125 52.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-63\"/>\n",
       "      <path d=\"M 54.890625 33.015625 \n",
       "L 54.890625 0 \n",
       "L 45.90625 0 \n",
       "L 45.90625 32.71875 \n",
       "Q 45.90625 40.484375 42.875 44.328125 \n",
       "Q 39.84375 48.1875 33.796875 48.1875 \n",
       "Q 26.515625 48.1875 22.3125 43.546875 \n",
       "Q 18.109375 38.921875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 75.984375 \n",
       "L 18.109375 75.984375 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.34375 51.125 25.703125 53.5625 \n",
       "Q 30.078125 56 35.796875 56 \n",
       "Q 45.21875 56 50.046875 50.171875 \n",
       "Q 54.890625 44.34375 54.890625 33.015625 \n",
       "z\n",
       "\" id=\"DejaVuSans-68\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(132.565625 174.876562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"m4536ecf47a\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m4536ecf47a\" y=\"138.525918\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0.25 -->\n",
       "      <g transform=\"translate(20.878125 142.325137)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m4536ecf47a\" y=\"114.091851\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0.30 -->\n",
       "      <defs>\n",
       "       <path d=\"M 40.578125 39.3125 \n",
       "Q 47.65625 37.796875 51.625 33 \n",
       "Q 55.609375 28.21875 55.609375 21.1875 \n",
       "Q 55.609375 10.40625 48.1875 4.484375 \n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \n",
       "Q 12.796875 0.390625 7.625 2.203125 \n",
       "L 7.625 11.71875 \n",
       "Q 11.71875 9.328125 16.59375 8.109375 \n",
       "Q 21.484375 6.890625 26.8125 6.890625 \n",
       "Q 36.078125 6.890625 40.9375 10.546875 \n",
       "Q 45.796875 14.203125 45.796875 21.1875 \n",
       "Q 45.796875 27.640625 41.28125 31.265625 \n",
       "Q 36.765625 34.90625 28.71875 34.90625 \n",
       "L 20.21875 34.90625 \n",
       "L 20.21875 43.015625 \n",
       "L 29.109375 43.015625 \n",
       "Q 36.375 43.015625 40.234375 45.921875 \n",
       "Q 44.09375 48.828125 44.09375 54.296875 \n",
       "Q 44.09375 59.90625 40.109375 62.90625 \n",
       "Q 36.140625 65.921875 28.71875 65.921875 \n",
       "Q 24.65625 65.921875 20.015625 65.03125 \n",
       "Q 15.375 64.15625 9.8125 62.3125 \n",
       "L 9.8125 71.09375 \n",
       "Q 15.4375 72.65625 20.34375 73.4375 \n",
       "Q 25.25 74.21875 29.59375 74.21875 \n",
       "Q 40.828125 74.21875 47.359375 69.109375 \n",
       "Q 53.90625 64.015625 53.90625 55.328125 \n",
       "Q 53.90625 49.265625 50.4375 45.09375 \n",
       "Q 46.96875 40.921875 40.578125 39.3125 \n",
       "z\n",
       "\" id=\"DejaVuSans-33\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(20.878125 117.89107)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m4536ecf47a\" y=\"89.657784\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0.35 -->\n",
       "      <g transform=\"translate(20.878125 93.457002)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m4536ecf47a\" y=\"65.223716\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.40 -->\n",
       "      <defs>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"DejaVuSans-34\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(20.878125 69.022935)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m4536ecf47a\" y=\"40.789649\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.45 -->\n",
       "      <g transform=\"translate(20.878125 44.588868)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m4536ecf47a\" y=\"16.355581\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 0.50 -->\n",
       "      <g transform=\"translate(20.878125 20.1548)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- loss -->\n",
       "     <defs>\n",
       "      <path d=\"M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-6c\"/>\n",
       "      <path d=\"M 44.28125 53.078125 \n",
       "L 44.28125 44.578125 \n",
       "Q 40.484375 46.53125 36.375 47.5 \n",
       "Q 32.28125 48.484375 27.875 48.484375 \n",
       "Q 21.1875 48.484375 17.84375 46.4375 \n",
       "Q 14.5 44.390625 14.5 40.28125 \n",
       "Q 14.5 37.15625 16.890625 35.375 \n",
       "Q 19.28125 33.59375 26.515625 31.984375 \n",
       "L 29.59375 31.296875 \n",
       "Q 39.15625 29.25 43.1875 25.515625 \n",
       "Q 47.21875 21.78125 47.21875 15.09375 \n",
       "Q 47.21875 7.46875 41.1875 3.015625 \n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \n",
       "Q 10.6875 0.296875 5.421875 2 \n",
       "L 5.421875 11.28125 \n",
       "Q 10.40625 8.6875 15.234375 7.390625 \n",
       "Q 20.0625 6.109375 24.8125 6.109375 \n",
       "Q 31.15625 6.109375 34.5625 8.28125 \n",
       "Q 37.984375 10.453125 37.984375 14.40625 \n",
       "Q 37.984375 18.0625 35.515625 20.015625 \n",
       "Q 33.0625 21.96875 24.703125 23.78125 \n",
       "L 21.578125 24.515625 \n",
       "Q 13.234375 26.265625 9.515625 29.90625 \n",
       "Q 5.8125 33.546875 5.8125 39.890625 \n",
       "Q 5.8125 47.609375 11.28125 51.796875 \n",
       "Q 16.75 56 26.8125 56 \n",
       "Q 31.78125 56 36.171875 55.265625 \n",
       "Q 40.578125 54.546875 44.28125 53.078125 \n",
       "z\n",
       "\" id=\"DejaVuSans-73\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(14.798437 88.307812)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-6c\"/>\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_12\">\n",
       "    <path clip-path=\"url(#p6cc29a0d5f)\" d=\"M 59.021023 16.877273 \n",
       "L 64.939205 84.046441 \n",
       "L 70.857386 139.314667 \n",
       "L 76.775568 128.987711 \n",
       "L 82.69375 138.880198 \n",
       "L 88.611932 138.030821 \n",
       "L 94.530114 134.526659 \n",
       "L 100.448295 136.909026 \n",
       "L 106.366477 140.422727 \n",
       "L 112.284659 138.116893 \n",
       "L 118.202841 138.310228 \n",
       "L 124.121023 135.719394 \n",
       "L 130.039205 134.022503 \n",
       "L 135.957386 135.932186 \n",
       "L 141.875568 136.501836 \n",
       "L 147.79375 133.968471 \n",
       "L 153.711932 138.876732 \n",
       "L 159.630114 139.432852 \n",
       "L 165.548295 137.569161 \n",
       "L 171.466477 134.319299 \n",
       "L 177.384659 134.893056 \n",
       "L 183.302841 134.376666 \n",
       "L 189.221023 140.114047 \n",
       "L 195.139205 136.307482 \n",
       "L 201.057386 139.885722 \n",
       "L 206.975568 134.381443 \n",
       "L 212.89375 133.470722 \n",
       "L 218.811932 128.70156 \n",
       "L 224.730114 131.182933 \n",
       "L 230.648295 122.549955 \n",
       "L 236.566477 138.072342 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 50.14375 146.6 \n",
       "L 50.14375 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 245.44375 146.6 \n",
       "L 245.44375 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 50.14375 146.6 \n",
       "L 245.44375 146.6 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 50.14375 10.7 \n",
       "L 245.44375 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p6cc29a0d5f\">\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"50.14375\" y=\"10.7\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "d2lt.train_ch7(sgd_momentum, init_momentum_states(),\n",
    "              {'lr': 0.02, 'momentum': 0.9}, features, labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可见目标函数值在后期迭代过程中的变化不够平滑。直觉上，10倍小批量梯度比2倍小批量梯度大了5倍，我们可以试着将学习率减小到原来的1/5。此时目标函数值在下降了一段时间后变化更加平滑。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 0.242837, 0.090186 sec per epoch\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 256.14375 184.15625\" width=\"256.14375pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 184.15625 \n",
       "L 256.14375 184.15625 \n",
       "L 256.14375 -0 \n",
       "L 0 -0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 50.14375 146.6 \n",
       "L 245.44375 146.6 \n",
       "L 245.44375 10.7 \n",
       "L 50.14375 10.7 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"m1db76a3abd\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"59.021023\" xlink:href=\"#m1db76a3abd\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0.0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-30\"/>\n",
       "       <path d=\"M 10.6875 12.40625 \n",
       "L 21 12.40625 \n",
       "L 21 0 \n",
       "L 10.6875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-2e\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(51.06946 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"103.407386\" xlink:href=\"#m1db76a3abd\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 0.5 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"DejaVuSans-35\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(95.455824 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"147.79375\" xlink:href=\"#m1db76a3abd\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 1.0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-31\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(139.842187 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"192.180114\" xlink:href=\"#m1db76a3abd\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 1.5 -->\n",
       "      <g transform=\"translate(184.228551 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"236.566477\" xlink:href=\"#m1db76a3abd\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 2.0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-32\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(228.614915 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <defs>\n",
       "      <path d=\"M 56.203125 29.59375 \n",
       "L 56.203125 25.203125 \n",
       "L 14.890625 25.203125 \n",
       "Q 15.484375 15.921875 20.484375 11.0625 \n",
       "Q 25.484375 6.203125 34.421875 6.203125 \n",
       "Q 39.59375 6.203125 44.453125 7.46875 \n",
       "Q 49.3125 8.734375 54.109375 11.28125 \n",
       "L 54.109375 2.78125 \n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \n",
       "Q 5.515625 13.8125 5.515625 26.8125 \n",
       "Q 5.515625 40.234375 12.765625 48.109375 \n",
       "Q 20.015625 56 32.328125 56 \n",
       "Q 43.359375 56 49.78125 48.890625 \n",
       "Q 56.203125 41.796875 56.203125 29.59375 \n",
       "z\n",
       "M 47.21875 32.234375 \n",
       "Q 47.125 39.59375 43.09375 43.984375 \n",
       "Q 39.0625 48.390625 32.421875 48.390625 \n",
       "Q 24.90625 48.390625 20.390625 44.140625 \n",
       "Q 15.875 39.890625 15.1875 32.171875 \n",
       "z\n",
       "\" id=\"DejaVuSans-65\"/>\n",
       "      <path d=\"M 18.109375 8.203125 \n",
       "L 18.109375 -20.796875 \n",
       "L 9.078125 -20.796875 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.390625 \n",
       "Q 20.953125 51.265625 25.265625 53.625 \n",
       "Q 29.59375 56 35.59375 56 \n",
       "Q 45.5625 56 51.78125 48.09375 \n",
       "Q 58.015625 40.1875 58.015625 27.296875 \n",
       "Q 58.015625 14.40625 51.78125 6.484375 \n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \n",
       "Q 20.953125 3.328125 18.109375 8.203125 \n",
       "z\n",
       "M 48.6875 27.296875 \n",
       "Q 48.6875 37.203125 44.609375 42.84375 \n",
       "Q 40.53125 48.484375 33.40625 48.484375 \n",
       "Q 26.265625 48.484375 22.1875 42.84375 \n",
       "Q 18.109375 37.203125 18.109375 27.296875 \n",
       "Q 18.109375 17.390625 22.1875 11.75 \n",
       "Q 26.265625 6.109375 33.40625 6.109375 \n",
       "Q 40.53125 6.109375 44.609375 11.75 \n",
       "Q 48.6875 17.390625 48.6875 27.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-70\"/>\n",
       "      <path d=\"M 30.609375 48.390625 \n",
       "Q 23.390625 48.390625 19.1875 42.75 \n",
       "Q 14.984375 37.109375 14.984375 27.296875 \n",
       "Q 14.984375 17.484375 19.15625 11.84375 \n",
       "Q 23.34375 6.203125 30.609375 6.203125 \n",
       "Q 37.796875 6.203125 41.984375 11.859375 \n",
       "Q 46.1875 17.53125 46.1875 27.296875 \n",
       "Q 46.1875 37.015625 41.984375 42.703125 \n",
       "Q 37.796875 48.390625 30.609375 48.390625 \n",
       "z\n",
       "M 30.609375 56 \n",
       "Q 42.328125 56 49.015625 48.375 \n",
       "Q 55.71875 40.765625 55.71875 27.296875 \n",
       "Q 55.71875 13.875 49.015625 6.21875 \n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \n",
       "Q 5.515625 13.875 5.515625 27.296875 \n",
       "Q 5.515625 40.765625 12.171875 48.375 \n",
       "Q 18.84375 56 30.609375 56 \n",
       "z\n",
       "\" id=\"DejaVuSans-6f\"/>\n",
       "      <path d=\"M 48.78125 52.59375 \n",
       "L 48.78125 44.1875 \n",
       "Q 44.96875 46.296875 41.140625 47.34375 \n",
       "Q 37.3125 48.390625 33.40625 48.390625 \n",
       "Q 24.65625 48.390625 19.8125 42.84375 \n",
       "Q 14.984375 37.3125 14.984375 27.296875 \n",
       "Q 14.984375 17.28125 19.8125 11.734375 \n",
       "Q 24.65625 6.203125 33.40625 6.203125 \n",
       "Q 37.3125 6.203125 41.140625 7.25 \n",
       "Q 44.96875 8.296875 48.78125 10.40625 \n",
       "L 48.78125 2.09375 \n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \n",
       "Q 5.515625 14.109375 5.515625 27.296875 \n",
       "Q 5.515625 40.671875 12.859375 48.328125 \n",
       "Q 20.21875 56 33.015625 56 \n",
       "Q 37.15625 56 41.109375 55.140625 \n",
       "Q 45.0625 54.296875 48.78125 52.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-63\"/>\n",
       "      <path d=\"M 54.890625 33.015625 \n",
       "L 54.890625 0 \n",
       "L 45.90625 0 \n",
       "L 45.90625 32.71875 \n",
       "Q 45.90625 40.484375 42.875 44.328125 \n",
       "Q 39.84375 48.1875 33.796875 48.1875 \n",
       "Q 26.515625 48.1875 22.3125 43.546875 \n",
       "Q 18.109375 38.921875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 75.984375 \n",
       "L 18.109375 75.984375 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.34375 51.125 25.703125 53.5625 \n",
       "Q 30.078125 56 35.796875 56 \n",
       "Q 45.21875 56 50.046875 50.171875 \n",
       "Q 54.890625 44.34375 54.890625 33.015625 \n",
       "z\n",
       "\" id=\"DejaVuSans-68\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(132.565625 174.876562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"m3f1582cfb7\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m3f1582cfb7\" y=\"136.817508\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0.25 -->\n",
       "      <g transform=\"translate(20.878125 140.616727)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m3f1582cfb7\" y=\"112.344643\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0.30 -->\n",
       "      <defs>\n",
       "       <path d=\"M 40.578125 39.3125 \n",
       "Q 47.65625 37.796875 51.625 33 \n",
       "Q 55.609375 28.21875 55.609375 21.1875 \n",
       "Q 55.609375 10.40625 48.1875 4.484375 \n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \n",
       "Q 12.796875 0.390625 7.625 2.203125 \n",
       "L 7.625 11.71875 \n",
       "Q 11.71875 9.328125 16.59375 8.109375 \n",
       "Q 21.484375 6.890625 26.8125 6.890625 \n",
       "Q 36.078125 6.890625 40.9375 10.546875 \n",
       "Q 45.796875 14.203125 45.796875 21.1875 \n",
       "Q 45.796875 27.640625 41.28125 31.265625 \n",
       "Q 36.765625 34.90625 28.71875 34.90625 \n",
       "L 20.21875 34.90625 \n",
       "L 20.21875 43.015625 \n",
       "L 29.109375 43.015625 \n",
       "Q 36.375 43.015625 40.234375 45.921875 \n",
       "Q 44.09375 48.828125 44.09375 54.296875 \n",
       "Q 44.09375 59.90625 40.109375 62.90625 \n",
       "Q 36.140625 65.921875 28.71875 65.921875 \n",
       "Q 24.65625 65.921875 20.015625 65.03125 \n",
       "Q 15.375 64.15625 9.8125 62.3125 \n",
       "L 9.8125 71.09375 \n",
       "Q 15.4375 72.65625 20.34375 73.4375 \n",
       "Q 25.25 74.21875 29.59375 74.21875 \n",
       "Q 40.828125 74.21875 47.359375 69.109375 \n",
       "Q 53.90625 64.015625 53.90625 55.328125 \n",
       "Q 53.90625 49.265625 50.4375 45.09375 \n",
       "Q 46.96875 40.921875 40.578125 39.3125 \n",
       "z\n",
       "\" id=\"DejaVuSans-33\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(20.878125 116.143861)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m3f1582cfb7\" y=\"87.871778\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0.35 -->\n",
       "      <g transform=\"translate(20.878125 91.670996)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m3f1582cfb7\" y=\"63.398912\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.40 -->\n",
       "      <defs>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"DejaVuSans-34\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(20.878125 67.198131)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m3f1582cfb7\" y=\"38.926047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.45 -->\n",
       "      <g transform=\"translate(20.878125 42.725266)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m3f1582cfb7\" y=\"14.453182\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 0.50 -->\n",
       "      <g transform=\"translate(20.878125 18.252401)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- loss -->\n",
       "     <defs>\n",
       "      <path d=\"M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-6c\"/>\n",
       "      <path d=\"M 44.28125 53.078125 \n",
       "L 44.28125 44.578125 \n",
       "Q 40.484375 46.53125 36.375 47.5 \n",
       "Q 32.28125 48.484375 27.875 48.484375 \n",
       "Q 21.1875 48.484375 17.84375 46.4375 \n",
       "Q 14.5 44.390625 14.5 40.28125 \n",
       "Q 14.5 37.15625 16.890625 35.375 \n",
       "Q 19.28125 33.59375 26.515625 31.984375 \n",
       "L 29.59375 31.296875 \n",
       "Q 39.15625 29.25 43.1875 25.515625 \n",
       "Q 47.21875 21.78125 47.21875 15.09375 \n",
       "Q 47.21875 7.46875 41.1875 3.015625 \n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \n",
       "Q 10.6875 0.296875 5.421875 2 \n",
       "L 5.421875 11.28125 \n",
       "Q 10.40625 8.6875 15.234375 7.390625 \n",
       "Q 20.0625 6.109375 24.8125 6.109375 \n",
       "Q 31.15625 6.109375 34.5625 8.28125 \n",
       "Q 37.984375 10.453125 37.984375 14.40625 \n",
       "Q 37.984375 18.0625 35.515625 20.015625 \n",
       "Q 33.0625 21.96875 24.703125 23.78125 \n",
       "L 21.578125 24.515625 \n",
       "Q 13.234375 26.265625 9.515625 29.90625 \n",
       "Q 5.8125 33.546875 5.8125 39.890625 \n",
       "Q 5.8125 47.609375 11.28125 51.796875 \n",
       "Q 16.75 56 26.8125 56 \n",
       "Q 31.78125 56 36.171875 55.265625 \n",
       "Q 40.578125 54.546875 44.28125 53.078125 \n",
       "z\n",
       "\" id=\"DejaVuSans-73\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(14.798437 88.307812)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-6c\"/>\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_12\">\n",
       "    <path clip-path=\"url(#p434c88f0a4)\" d=\"M 59.021023 16.877273 \n",
       "L 64.939205 55.154065 \n",
       "L 70.857386 87.992881 \n",
       "L 76.775568 112.156917 \n",
       "L 82.69375 125.247522 \n",
       "L 88.611932 132.437699 \n",
       "L 94.530114 134.566666 \n",
       "L 100.448295 134.376584 \n",
       "L 106.366477 134.832849 \n",
       "L 112.284659 135.57111 \n",
       "L 118.202841 136.470163 \n",
       "L 124.121023 137.724963 \n",
       "L 130.039205 137.450692 \n",
       "L 135.957386 139.246821 \n",
       "L 141.875568 138.365644 \n",
       "L 147.79375 137.783318 \n",
       "L 153.711932 136.7369 \n",
       "L 159.630114 136.416264 \n",
       "L 165.548295 139.883936 \n",
       "L 171.466477 139.956338 \n",
       "L 177.384659 138.232852 \n",
       "L 183.302841 135.765554 \n",
       "L 189.221023 138.344165 \n",
       "L 195.139205 140.132695 \n",
       "L 201.057386 139.541718 \n",
       "L 206.975568 139.598936 \n",
       "L 212.89375 140.330691 \n",
       "L 218.811932 140.422727 \n",
       "L 224.730114 139.303754 \n",
       "L 230.648295 140.343031 \n",
       "L 236.566477 140.32339 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 50.14375 146.6 \n",
       "L 50.14375 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 245.44375 146.6 \n",
       "L 245.44375 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 50.14375 146.6 \n",
       "L 245.44375 146.6 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 50.14375 10.7 \n",
       "L 245.44375 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p434c88f0a4\">\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"50.14375\" y=\"10.7\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "d2lt.train_ch7(sgd_momentum, init_momentum_states(),\n",
    "              {'lr': 0.004, 'momentum': 0.9}, features, labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 简洁实现\n",
    "\n",
    "在PyTorch中，只需要在`Optimizer`实例中通过`momentum`来指定动量超参数即可使用动量法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "9"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 0.242904, 0.060073 sec per epoch\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\n",
       "<svg height=\"184.15625pt\" version=\"1.1\" viewBox=\"0 0 256.14375 184.15625\" width=\"256.14375pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 184.15625 \n",
       "L 256.14375 184.15625 \n",
       "L 256.14375 -0 \n",
       "L 0 -0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 50.14375 146.6 \n",
       "L 245.44375 146.6 \n",
       "L 245.44375 10.7 \n",
       "L 50.14375 10.7 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"m1026e30fdc\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"59.021023\" xlink:href=\"#m1026e30fdc\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0.0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-30\"/>\n",
       "       <path d=\"M 10.6875 12.40625 \n",
       "L 21 12.40625 \n",
       "L 21 0 \n",
       "L 10.6875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-2e\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(51.06946 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"103.407386\" xlink:href=\"#m1026e30fdc\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 0.5 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"DejaVuSans-35\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(95.455824 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"147.79375\" xlink:href=\"#m1026e30fdc\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 1.0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-31\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(139.842187 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"192.180114\" xlink:href=\"#m1026e30fdc\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 1.5 -->\n",
       "      <g transform=\"translate(184.228551 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"236.566477\" xlink:href=\"#m1026e30fdc\" y=\"146.6\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 2.0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-32\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(228.614915 161.198437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <defs>\n",
       "      <path d=\"M 56.203125 29.59375 \n",
       "L 56.203125 25.203125 \n",
       "L 14.890625 25.203125 \n",
       "Q 15.484375 15.921875 20.484375 11.0625 \n",
       "Q 25.484375 6.203125 34.421875 6.203125 \n",
       "Q 39.59375 6.203125 44.453125 7.46875 \n",
       "Q 49.3125 8.734375 54.109375 11.28125 \n",
       "L 54.109375 2.78125 \n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \n",
       "Q 5.515625 13.8125 5.515625 26.8125 \n",
       "Q 5.515625 40.234375 12.765625 48.109375 \n",
       "Q 20.015625 56 32.328125 56 \n",
       "Q 43.359375 56 49.78125 48.890625 \n",
       "Q 56.203125 41.796875 56.203125 29.59375 \n",
       "z\n",
       "M 47.21875 32.234375 \n",
       "Q 47.125 39.59375 43.09375 43.984375 \n",
       "Q 39.0625 48.390625 32.421875 48.390625 \n",
       "Q 24.90625 48.390625 20.390625 44.140625 \n",
       "Q 15.875 39.890625 15.1875 32.171875 \n",
       "z\n",
       "\" id=\"DejaVuSans-65\"/>\n",
       "      <path d=\"M 18.109375 8.203125 \n",
       "L 18.109375 -20.796875 \n",
       "L 9.078125 -20.796875 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.390625 \n",
       "Q 20.953125 51.265625 25.265625 53.625 \n",
       "Q 29.59375 56 35.59375 56 \n",
       "Q 45.5625 56 51.78125 48.09375 \n",
       "Q 58.015625 40.1875 58.015625 27.296875 \n",
       "Q 58.015625 14.40625 51.78125 6.484375 \n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \n",
       "Q 20.953125 3.328125 18.109375 8.203125 \n",
       "z\n",
       "M 48.6875 27.296875 \n",
       "Q 48.6875 37.203125 44.609375 42.84375 \n",
       "Q 40.53125 48.484375 33.40625 48.484375 \n",
       "Q 26.265625 48.484375 22.1875 42.84375 \n",
       "Q 18.109375 37.203125 18.109375 27.296875 \n",
       "Q 18.109375 17.390625 22.1875 11.75 \n",
       "Q 26.265625 6.109375 33.40625 6.109375 \n",
       "Q 40.53125 6.109375 44.609375 11.75 \n",
       "Q 48.6875 17.390625 48.6875 27.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-70\"/>\n",
       "      <path d=\"M 30.609375 48.390625 \n",
       "Q 23.390625 48.390625 19.1875 42.75 \n",
       "Q 14.984375 37.109375 14.984375 27.296875 \n",
       "Q 14.984375 17.484375 19.15625 11.84375 \n",
       "Q 23.34375 6.203125 30.609375 6.203125 \n",
       "Q 37.796875 6.203125 41.984375 11.859375 \n",
       "Q 46.1875 17.53125 46.1875 27.296875 \n",
       "Q 46.1875 37.015625 41.984375 42.703125 \n",
       "Q 37.796875 48.390625 30.609375 48.390625 \n",
       "z\n",
       "M 30.609375 56 \n",
       "Q 42.328125 56 49.015625 48.375 \n",
       "Q 55.71875 40.765625 55.71875 27.296875 \n",
       "Q 55.71875 13.875 49.015625 6.21875 \n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \n",
       "Q 5.515625 13.875 5.515625 27.296875 \n",
       "Q 5.515625 40.765625 12.171875 48.375 \n",
       "Q 18.84375 56 30.609375 56 \n",
       "z\n",
       "\" id=\"DejaVuSans-6f\"/>\n",
       "      <path d=\"M 48.78125 52.59375 \n",
       "L 48.78125 44.1875 \n",
       "Q 44.96875 46.296875 41.140625 47.34375 \n",
       "Q 37.3125 48.390625 33.40625 48.390625 \n",
       "Q 24.65625 48.390625 19.8125 42.84375 \n",
       "Q 14.984375 37.3125 14.984375 27.296875 \n",
       "Q 14.984375 17.28125 19.8125 11.734375 \n",
       "Q 24.65625 6.203125 33.40625 6.203125 \n",
       "Q 37.3125 6.203125 41.140625 7.25 \n",
       "Q 44.96875 8.296875 48.78125 10.40625 \n",
       "L 48.78125 2.09375 \n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \n",
       "Q 5.515625 14.109375 5.515625 27.296875 \n",
       "Q 5.515625 40.671875 12.859375 48.328125 \n",
       "Q 20.21875 56 33.015625 56 \n",
       "Q 37.15625 56 41.109375 55.140625 \n",
       "Q 45.0625 54.296875 48.78125 52.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-63\"/>\n",
       "      <path d=\"M 54.890625 33.015625 \n",
       "L 54.890625 0 \n",
       "L 45.90625 0 \n",
       "L 45.90625 32.71875 \n",
       "Q 45.90625 40.484375 42.875 44.328125 \n",
       "Q 39.84375 48.1875 33.796875 48.1875 \n",
       "Q 26.515625 48.1875 22.3125 43.546875 \n",
       "Q 18.109375 38.921875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 75.984375 \n",
       "L 18.109375 75.984375 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.34375 51.125 25.703125 53.5625 \n",
       "Q 30.078125 56 35.796875 56 \n",
       "Q 45.21875 56 50.046875 50.171875 \n",
       "Q 54.890625 44.34375 54.890625 33.015625 \n",
       "z\n",
       "\" id=\"DejaVuSans-68\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(132.565625 174.876562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"m50f8bf877e\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m50f8bf877e\" y=\"136.611371\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0.25 -->\n",
       "      <g transform=\"translate(20.878125 140.41059)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m50f8bf877e\" y=\"112.27518\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0.30 -->\n",
       "      <defs>\n",
       "       <path d=\"M 40.578125 39.3125 \n",
       "Q 47.65625 37.796875 51.625 33 \n",
       "Q 55.609375 28.21875 55.609375 21.1875 \n",
       "Q 55.609375 10.40625 48.1875 4.484375 \n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \n",
       "Q 12.796875 0.390625 7.625 2.203125 \n",
       "L 7.625 11.71875 \n",
       "Q 11.71875 9.328125 16.59375 8.109375 \n",
       "Q 21.484375 6.890625 26.8125 6.890625 \n",
       "Q 36.078125 6.890625 40.9375 10.546875 \n",
       "Q 45.796875 14.203125 45.796875 21.1875 \n",
       "Q 45.796875 27.640625 41.28125 31.265625 \n",
       "Q 36.765625 34.90625 28.71875 34.90625 \n",
       "L 20.21875 34.90625 \n",
       "L 20.21875 43.015625 \n",
       "L 29.109375 43.015625 \n",
       "Q 36.375 43.015625 40.234375 45.921875 \n",
       "Q 44.09375 48.828125 44.09375 54.296875 \n",
       "Q 44.09375 59.90625 40.109375 62.90625 \n",
       "Q 36.140625 65.921875 28.71875 65.921875 \n",
       "Q 24.65625 65.921875 20.015625 65.03125 \n",
       "Q 15.375 64.15625 9.8125 62.3125 \n",
       "L 9.8125 71.09375 \n",
       "Q 15.4375 72.65625 20.34375 73.4375 \n",
       "Q 25.25 74.21875 29.59375 74.21875 \n",
       "Q 40.828125 74.21875 47.359375 69.109375 \n",
       "Q 53.90625 64.015625 53.90625 55.328125 \n",
       "Q 53.90625 49.265625 50.4375 45.09375 \n",
       "Q 46.96875 40.921875 40.578125 39.3125 \n",
       "z\n",
       "\" id=\"DejaVuSans-33\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(20.878125 116.074399)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m50f8bf877e\" y=\"87.938989\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0.35 -->\n",
       "      <g transform=\"translate(20.878125 91.738208)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m50f8bf877e\" y=\"63.602799\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.40 -->\n",
       "      <defs>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"DejaVuSans-34\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(20.878125 67.402017)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m50f8bf877e\" y=\"39.266608\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.45 -->\n",
       "      <g transform=\"translate(20.878125 43.065827)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#m50f8bf877e\" y=\"14.930417\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 0.50 -->\n",
       "      <g transform=\"translate(20.878125 18.729636)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- loss -->\n",
       "     <defs>\n",
       "      <path d=\"M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-6c\"/>\n",
       "      <path d=\"M 44.28125 53.078125 \n",
       "L 44.28125 44.578125 \n",
       "Q 40.484375 46.53125 36.375 47.5 \n",
       "Q 32.28125 48.484375 27.875 48.484375 \n",
       "Q 21.1875 48.484375 17.84375 46.4375 \n",
       "Q 14.5 44.390625 14.5 40.28125 \n",
       "Q 14.5 37.15625 16.890625 35.375 \n",
       "Q 19.28125 33.59375 26.515625 31.984375 \n",
       "L 29.59375 31.296875 \n",
       "Q 39.15625 29.25 43.1875 25.515625 \n",
       "Q 47.21875 21.78125 47.21875 15.09375 \n",
       "Q 47.21875 7.46875 41.1875 3.015625 \n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \n",
       "Q 10.6875 0.296875 5.421875 2 \n",
       "L 5.421875 11.28125 \n",
       "Q 10.40625 8.6875 15.234375 7.390625 \n",
       "Q 20.0625 6.109375 24.8125 6.109375 \n",
       "Q 31.15625 6.109375 34.5625 8.28125 \n",
       "Q 37.984375 10.453125 37.984375 14.40625 \n",
       "Q 37.984375 18.0625 35.515625 20.015625 \n",
       "Q 33.0625 21.96875 24.703125 23.78125 \n",
       "L 21.578125 24.515625 \n",
       "Q 13.234375 26.265625 9.515625 29.90625 \n",
       "Q 5.8125 33.546875 5.8125 39.890625 \n",
       "Q 5.8125 47.609375 11.28125 51.796875 \n",
       "Q 16.75 56 26.8125 56 \n",
       "Q 31.78125 56 36.171875 55.265625 \n",
       "Q 40.578125 54.546875 44.28125 53.078125 \n",
       "z\n",
       "\" id=\"DejaVuSans-73\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(14.798437 88.307812)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-6c\"/>\n",
       "      <use x=\"27.783203\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"88.964844\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "      <use x=\"141.064453\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_12\">\n",
       "    <path clip-path=\"url(#pe64a4b5e6e)\" d=\"M 59.021023 16.877273 \n",
       "L 64.939205 41.16914 \n",
       "L 70.857386 72.713222 \n",
       "L 76.775568 101.399966 \n",
       "L 82.69375 119.186566 \n",
       "L 88.611932 125.281995 \n",
       "L 94.530114 131.523755 \n",
       "L 100.448295 134.797603 \n",
       "L 106.366477 136.640527 \n",
       "L 112.284659 137.235651 \n",
       "L 118.202841 136.869387 \n",
       "L 124.121023 136.177729 \n",
       "L 130.039205 137.409463 \n",
       "L 135.957386 137.190459 \n",
       "L 141.875568 137.647064 \n",
       "L 147.79375 138.823547 \n",
       "L 153.711932 138.830451 \n",
       "L 159.630114 137.877491 \n",
       "L 165.548295 137.311443 \n",
       "L 171.466477 139.097831 \n",
       "L 177.384659 140.082834 \n",
       "L 183.302841 140.063332 \n",
       "L 189.221023 139.81148 \n",
       "L 195.139205 139.014483 \n",
       "L 201.057386 139.487891 \n",
       "L 206.975568 140.248465 \n",
       "L 212.89375 140.422727 \n",
       "L 218.811932 140.325932 \n",
       "L 224.730114 140.368259 \n",
       "L 230.648295 139.868371 \n",
       "L 236.566477 140.065254 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 50.14375 146.6 \n",
       "L 50.14375 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 245.44375 146.6 \n",
       "L 245.44375 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 50.14375 146.6 \n",
       "L 245.44375 146.6 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 50.14375 10.7 \n",
       "L 245.44375 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pe64a4b5e6e\">\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"50.14375\" y=\"10.7\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "d2lt.train_nn_ch7(optim.SGD, {'lr': 0.004, 'momentum': 0.9},\n",
    "                    features, labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 小结\n",
    "\n",
    "* 动量法使用了指数加权移动平均的思想。它将过去时间步的梯度做了加权平均，且权重按时间步指数衰减。\n",
    "* 动量法使得相邻时间步的自变量更新在方向上更加一致。\n",
    "\n",
    "## 练习\n",
    "\n",
    "* 使用其他动量超参数和学习率的组合，观察并分析实验结果。\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## 扫码直达[讨论区](https://discuss.gluon.ai/t/topic/1879)\n",
    "\n",
    "![](../img/qr_momentum.svg)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:pytorch]",
   "language": "python",
   "name": "conda-env-pytorch-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
